Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Sensor data logging on Raspberry PI and SD card life...
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
VinzC
Advocate
Advocate


Joined: 17 Apr 2004
Posts: 4930
Location: Dark side of the mood

PostPosted: Wed Apr 04, 2018 10:34 am    Post subject: Sensor data logging on Raspberry PI and SD card life... Reply with quote

Hi all.

I'm using my Raspberry Pi as a testing gear. I have a script that collects data from a sensor and writes it into a CSV file. For some reason, data logging is done every second as follows:
  1. open the CSV file,
  2. write the few line characters from the sensor,
  3. close the file
Now I'm afraid this might speed up the SD card wearing out and reduce the card's life span. What tickles is that the sensor data is stored on the same partition as the root filesystem, which is mounted read-write, of course.

I'm considering a temporary filesystem to store the data and then copy to long-term storage after data collection is done, for example. I also considered some filesystem caching using overlayfs... but all of this seems a bit overkill, I'd say. Ideally I'd tweak the filesystem to buffer the sensor data for a couple of device blocks and then write them to the SD card.

I guess ext4 (or F2FS, maybe) might just have what it takes, right? Just that I have no clue what parameter to tune. Any clue?

EDIT: To get an idea about the volume of data, the file size is about one megabyte after eight hours.

Thanks in advance.
_________________
Dealing with computers is the best way to get sucked into more troubles than any sane person can cope with!
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
Back to top
View user's profile Send private message
bbgermany
Veteran
Veteran


Joined: 21 Feb 2005
Posts: 1771
Location: Oranienburg/Germany

PostPosted: Wed Apr 04, 2018 12:29 pm    Post subject: Reply with quote

Hi,

Which version of the rpi are you using? rpi2 with BCM2837 or rpi3 can boot from usb or via network. So you could get around this issue ;)

greets, bb
_________________
1st: i5-4570, 16GB, 1.75TB
2nd: i5-4570, 8GB, 620GB
3rd: i5-4570, 16GB, 10TB
4th: Asus N61VN, 8GB, 120GB
5th: Cubietruck, 2GB, 160GB + NFS
6th: C2D T7200, 2GB, 16GB USB + NFS
7th: RPi3 1GB, 64GB USB
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 6393
Location: Saint Amant, Acadiana

PostPosted: Wed Apr 04, 2018 1:19 pm    Post subject: Reply with quote

Even RPi1 can use USB as external storage. Or use NFS if your device is networked.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
VinzC
Advocate
Advocate


Joined: 17 Apr 2004
Posts: 4930
Location: Dark side of the mood

PostPosted: Wed Apr 04, 2018 1:51 pm    Post subject: Reply with quote

Thanks for your suggestions, guys but I don't want to use anything but the SD card. I only want to know more of how (and if) I can tune the filesystem to not cause "too many" block writes^Werases, which IMHO would reduce the card lifespan. So although what I want to know is irrelevant of the Pi version, mine is a Pi model 2B.
_________________
Dealing with computers is the best way to get sucked into more troubles than any sane person can cope with!
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...


Last edited by VinzC on Wed Apr 04, 2018 2:30 pm; edited 1 time in total
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 537
Location: Frankfurt, Germany

PostPosted: Wed Apr 04, 2018 2:21 pm    Post subject: Reply with quote

vm.dirty_writeback_centisecs = 1500

/usr/src/linux/Documentation/sysctl/vm.txt wrote:
dirty_writeback_centisecs

The kernel flusher threads will periodically wake up and write `old' data out to disk. This tunable expresses the interval between those wakeups, in 100'ths of a second.

Setting this to zero disables periodic writeback altogether.
Back to top
View user's profile Send private message
VinzC
Advocate
Advocate


Joined: 17 Apr 2004
Posts: 4930
Location: Dark side of the mood

PostPosted: Wed Apr 04, 2018 3:08 pm    Post subject: Reply with quote

Thanks mike155, you've answered my question. However — my bad — I should have stated «block erases» instead of «block writes» as the former more than the latter is responsible for the wearing out effect. Meanwhile I've sought the interweb and from what I've learnt I think I'm going to stick with F2FS and its defaults, storing my sensors' data on a separate partition.

From what I've read on StackOverflow invalidating two 4KB block every second would make my 8GB SD card last about 30 years... extrapolating from the example given by DrV, of course. If F2FS can increase life expectancy by ~70% compared against ext4, then there's no reason not to use it.
_________________
Dealing with computers is the best way to get sucked into more troubles than any sane person can cope with!
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
Back to top
View user's profile Send private message
mike155
Guru
Guru


Joined: 17 Sep 2010
Posts: 537
Location: Frankfurt, Germany

PostPosted: Wed Apr 04, 2018 3:30 pm    Post subject: Reply with quote

VinzC,

here is what I would do to reduce the number of writes to the SD card:

1) I would let the data collector (which touches the output file every second) write to a data file on a tmpfs file system

2) I would write a small cronjob that runs every 5 minutes and that appends records, which were written to the output file on the tmpfs during the last minutes, to a data file, which is stored on the SD card. Something like:
Code:
len1 = length of data file on SD card
len2 = length of data file on tmpfs

if ( len2 > len1 ) {
    dd if=<data file on tmpfs> bs=1 skip=<len1> >> <data file on sd card>
}
Back to top
View user's profile Send private message
VinzC
Advocate
Advocate


Joined: 17 Apr 2004
Posts: 4930
Location: Dark side of the mood

PostPosted: Wed Apr 04, 2018 4:41 pm    Post subject: Reply with quote

Thanks mike. I think I'd implement this bucket-write feature in my Python script to avoid any potential side effect like writing to the data file while reading it. That said I believe F2FS is smart enough to do better than any software trick I'd implement — I'd need thorough testing to demonstrate that, of course.
_________________
Dealing with computers is the best way to get sucked into more troubles than any sane person can cope with!
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
Back to top
View user's profile Send private message
VinzC
Advocate
Advocate


Joined: 17 Apr 2004
Posts: 4930
Location: Dark side of the mood

PostPosted: Thu Apr 12, 2018 8:30 pm    Post subject: Reply with quote

I've finally found a way: I've used overlayfs so that the data are stored in a file on a tmpfs, which is mounted on /tmp/sensors. OverlayFS needs a permanent storage, which I mounted under /mnt/sensors as F2FS. The data file is then transparently moved from the temporary storage in /tmp/sensors to /mnt/sensors. Done.
_________________
Dealing with computers is the best way to get sucked into more troubles than any sane person can cope with!
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum