Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
thoughts about the impact of SSD usage on code quality
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Off the Wall
View previous topic :: View next topic  
Author Message
e3k
Apprentice
Apprentice


Joined: 01 Oct 2007
Posts: 200
Location: Slovakia

PostPosted: Fri Oct 11, 2013 12:15 pm    Post subject: thoughts about the impact of SSD usage on code quality Reply with quote

I run a system with XFCE and was wondering why my HDD sporadically starts working for a few minutes with no apparent reason. After installing iotop i figured out that the responsible process is tumlerd. Further i have found a year old bug that describes this behavior after deleting a .jpg for example.

Do developers now use SSDs and cant see that they program inefficient code because on SSDs random access is fast? If yes then is it only a matter of time when our software becomes far more inefficient than before the SSDs era and the advantages of SSDs will be not that significant as they were at the beginning?
_________________
---__(o)__---
  | |
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Fri Oct 11, 2013 1:51 pm    Post subject: Reply with quote

Could be a few reasons, here are some vids to help explain:

Linky 1
Linky 2
Back to top
View user's profile Send private message
sikpuppy
n00b
n00b


Joined: 12 Jun 2012
Posts: 34
Location: Central Coast, NSW

PostPosted: Fri Oct 11, 2013 1:58 pm    Post subject: Reply with quote

Tumbler is a waste of time, unless you really need each media file to have a thumbnail. Personally I would --depclean tumbler, since it slows down the Thunar file manager when opening a large folder full of video or picture files that have not already been thumbnailed. Also delete ~.thumbnails. Problem avoided, perhaps not solved.
Back to top
View user's profile Send private message
e3k
Apprentice
Apprentice


Joined: 01 Oct 2007
Posts: 200
Location: Slovakia

PostPosted: Fri Oct 11, 2013 2:41 pm    Post subject: Reply with quote

@Bigun: thanks but the thought was more like: Developers having an SSD on PC are prone to build ineffective code for PCs with HDDs.
@sikpuppy: thumbnails are really helpful when organizing one of the many photo albums of my father in law just with an file manager.
_________________
---__(o)__---
  | |
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Fri Oct 11, 2013 3:00 pm    Post subject: Reply with quote

e3k wrote:
@Bigun: thanks but the thought was more like: Developers having an SSD on PC are prone to build ineffective code for PCs with HDDs.


Ahh, sorry. I misunderstood.
Back to top
View user's profile Send private message
Prenj
n00b
n00b


Joined: 20 Nov 2011
Posts: 13

PostPosted: Fri Oct 11, 2013 3:04 pm    Post subject: Reply with quote

I blame the culture of superficiality and low attention span. It's now how good is the code you write, but how you look while doing it! :lol:
_________________
“If You Meet the Buddha on the Road, Kill Him”
Back to top
View user's profile Send private message
Dr.Willy
Guru
Guru


Joined: 15 Jul 2007
Posts: 346
Location: NRW, Germany

PostPosted: Fri Oct 11, 2013 3:14 pm    Post subject: Reply with quote

e3k wrote:
@Bigun: thanks but the thought was more like: Developers having an SSD on PC are prone to build ineffective code for PCs with HDDs.

Yes.
Or do you know anyone still optimizing their code for punch-card reading?
Back to top
View user's profile Send private message
sikpuppy
n00b
n00b


Joined: 12 Jun 2012
Posts: 34
Location: Central Coast, NSW

PostPosted: Fri Oct 11, 2013 3:37 pm    Post subject: Reply with quote

e3k wrote:
@Bigun: thanks but the thought was more like: Developers having an SSD on PC are prone to build ineffective code for PCs with HDDs.
@sikpuppy: thumbnails are really helpful when organizing one of the many photo albums of my father in law just with an file manager.
Using a file manager is a slow and painful way of organising a lot of photos. Even using the simple image viewer Ristretto that xfce promotes on their site has to be a better bet. It includes thumbnails in the side panel of the main screen.
Back to top
View user's profile Send private message
e3k
Apprentice
Apprentice


Joined: 01 Oct 2007
Posts: 200
Location: Slovakia

PostPosted: Fri Oct 11, 2013 7:22 pm    Post subject: Reply with quote

Dr.Willy wrote:
e3k wrote:
@Bigun: thanks but the thought was more like: Developers having an SSD on PC are prone to build ineffective code for PCs with HDDs.

Yes.
Or do you know anyone still optimizing their code for punch-card reading?

sorry but when i delete one jpeg and i am a thumbd i should look for the corresponding thumbnail and delete that to. that should not take more than a few mseconds even with an hdd.

@sikpuppy: no thats not the way how we work. we just create a few folders and put the photos there. no need for something complicated. i dont want to bother my father in law with ristreto when he looks at the photos using his xp laptop. i am happy he handled his local software there.
_________________
---__(o)__---
  | |
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 4010
Location: USA

PostPosted: Fri Oct 11, 2013 9:45 pm    Post subject: Reply with quote

This is hardly a new problem. Ever since RAM became immense and cpus got really fast, people have been tossing software problems at the hardware and letting it figure it out. Even things like virtual memory/swap and caches are throwing software problems at the hardware/OS. Now SSDs are yet another crutch.

"It's now how good is the code you write, but how you look while doing it!" - there's some reality into this, but more of how fast you can dump the code out and put it on the market. Doesn't matter if it's slow or buggy, unpackaged code not on the shelf doesn't produce profits...

Sad.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
sikpuppy
n00b
n00b


Joined: 12 Jun 2012
Posts: 34
Location: Central Coast, NSW

PostPosted: Sat Oct 12, 2013 3:47 am    Post subject: Reply with quote

Well, if the problem is technophobia then it's a WON'T FIX from me then. :P
Back to top
View user's profile Send private message
NiHaoMike
n00b
n00b


Joined: 29 May 2013
Posts: 5
Location: Austin, TX

PostPosted: Sat Oct 12, 2013 4:32 am    Post subject: Reply with quote

Why spend hours hand tuning assembly when you can whack together something in Python in a few minutes that runs with acceptable speed on a modern CPU? Computing power is cheap nowadays, but time isn't.

That's also why hobbyists generally buy the higher end microcontrollers. For a one off, the extra cost is very quickly returned in saved time. Only in high volume production does it start making sense to spend the time.
_________________
Core i7-3930k
DX79SI
16GB 1600 DDR3
GTX 970 4GB
120mm Delta fan with Cindy Wu sensorless DSP drive
900W Lainey Schmidt digital power system
128GB Samsung 840 Pro SSD (/)
2x 1TB HDD RAID 0 (/bulk)
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Sat Oct 12, 2013 4:59 am    Post subject: Reply with quote

NiHaoMike wrote:
Computing power is cheap nowadays, but time isn't.
For the benefit of the environment, a computing tax on electricity is needed. The lives of millions shouldn't be threatened just so you can write inefficient code. :P Sorry, couldn't resist.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
NiHaoMike
n00b
n00b


Joined: 29 May 2013
Posts: 5
Location: Austin, TX

PostPosted: Sat Oct 12, 2013 5:17 am    Post subject: Reply with quote

pjp wrote:
NiHaoMike wrote:
Computing power is cheap nowadays, but time isn't.
For the benefit of the environment, a computing tax on electricity is needed. The lives of millions shouldn't be threatened just so you can write inefficient code. :P Sorry, couldn't resist.

CPU efficiency is improving enough to outstrip inefficiency in code. That's especially true of mobile devices. In fact, it turns out that in a lot of mobile devices, the most effective efficiency tweaks in code don't really involve the CPU. For example, when displaying text, render it as full black on full white (other way around for OLED) and turn down the brightness to get the power savings. (A lot of consoles default to gray on black - just plain awful on LCDs.)
_________________
Core i7-3930k
DX79SI
16GB 1600 DDR3
GTX 970 4GB
120mm Delta fan with Cindy Wu sensorless DSP drive
900W Lainey Schmidt digital power system
128GB Samsung 840 Pro SSD (/)
2x 1TB HDD RAID 0 (/bulk)
Back to top
View user's profile Send private message
Prenj
n00b
n00b


Joined: 20 Nov 2011
Posts: 13

PostPosted: Sat Oct 12, 2013 6:46 am    Post subject: Reply with quote

NiHaoMike wrote:
pjp wrote:
NiHaoMike wrote:
Computing power is cheap nowadays, but time isn't.
For the benefit of the environment, a computing tax on electricity is needed. The lives of millions shouldn't be threatened just so you can write inefficient code. :P Sorry, couldn't resist.

CPU efficiency is improving enough to outstrip inefficiency in code. That's especially true of mobile devices. In fact, it turns out that in a lot of mobile devices, the most effective efficiency tweaks in code don't really involve the CPU. For example, when displaying text, render it as full black on full white (other way around for OLED) and turn down the brightness to get the power savings. (A lot of consoles default to gray on black - just plain awful on LCDs.)

Actually that's wrong. It used to be so couple of years ago-

The progress today is greater number of cores (and power efficiency), and to take advantage of that, you have to write better code. Even your python example runs on single core, if you don't make sure it does otherwise. And so on.

As I said, less Linkedin fuckery, more coding.
_________________
“If You Meet the Buddha on the Road, Kill Him”
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1102
Location: Rep. of Ireland

PostPosted: Sat Oct 12, 2013 9:32 am    Post subject: Reply with quote

Actually, if your software's efficiency factor is well below than the number of CPU cores (and a *metric fucktonne* of software easily falls under this), then adding multithreading (assuming your software lends itself to parallelism at all) is a poor first choice because you're not going to get the same performance and power saving as just cleaning up your sloppy code and making it efficient. Once you get that mess cleaned up, then you can start to think about throwing more cores at it.

It's inevitable that sloppy programming practices is going to bite us on the ass the moment Moore's law ceases to be applicable as physical constraints bars the amount of hardware we can throw at our failure to programme properly. The rise of the multicore machine was the first step towards that inevitability of being forced into more coding work to improve performance, but it's just patching up around inherent software suckiness, most will do the bare minimum to break their software up into multicore friendly chunks. The shift towards commonplace GPGPU co-processing will be the next. But they're still not getting the message. Throwing more computation resources to make your shitty half-assed Java project work isn't going to cut it anymore.
_________________
juniper wrote:
you experience political reality dilation when travelling at american political speeds. it's in einstein's formulas. it's not their fault.
Back to top
View user's profile Send private message
ryao
Developer
Developer


Joined: 27 Feb 2012
Posts: 123

PostPosted: Sat Oct 12, 2013 3:08 pm    Post subject: Reply with quote

I am involved with ZFS development, so I can talk about the performance work being done in ZFS and whether it benefits one form of storage over another; specifically, traditional rotational magnetic storage and NAND-based solid state storage. ZFS stands for "Zettabyte FileSystem" and true to its name, it is designed to store enormous quantities of data. At this time, mechanical disks are the only feasible way of storing large quantities of data, which requires us to pay attention to them.

ZFS was among the first filesystems to integrate explicit support for flash storage. This was done by extending the concepts of ARC and ZIL via L2ARC and SLOG devices. The L2ARC devices act as a second level read cache between main memory and the disk. The SLOG devices move the ZFS Intent Log (ZIL) off the main disks so that synchronous writes are converted into asynchronous writes and processed almost instantaneously. NAND-flash storage can be used in place of the mechanical hard drives and this will be quite fast, but nothing has been done to compromise performance for mechanical hard drives in favor of them. Many of the features designed to make ZFS fast on mechanical disks (e.g. ZIL, ARC, variable stripe size, transparent compression) also benefit solid state disks and people using solid state disks for storage will have a positive experience.

With that said, there are improvements in the queue for ZFSOnLinux 0.6.3 that should yield generic performance improvements that benefit both forms of storage:

https://github.com/ryao/zfs/commit/58b15712bc5dd8fadd669067bc98db6437f24a45
https://github.com/ryao/zfs/commit/2c5ea88556e56957b1fbe3a943e3fb25f8888aae
https://github.com/ryao/zfs/commit/2c128513ff3022f31280d0175b38837c72cfdc04
https://github.com/ryao/zfs/commit/70eed34f9fca804734fa8388a37a2e45c908d6c8

In a related vein, completely opposite design decisions made in Solaris and Linux cause ZFS to deadlock on 32-bit Linux (with any amount of memory). I am working on patches to fix that. When they are ready, I expect them to make ZFS run smoothly on the Raspberry Pi and also on other 32-bit systems with as little as 128MB of RAM (and possibly less). Having more resources for development is not going to stop ZFS from scaling down to older systems. However, I do not expect to see ZFS scale down below 32MB of RAM and/or below 64MB storage devices. The former likely would not leave enough memory available for a useful userland while the latter is a minimum requirement that was decided when ZFS was created. Note that data deduplication will still require large amounts of RAM to run well.

e3k wrote:
I run a system with XFCE and was wondering why my HDD sporadically starts working for a few minutes with no apparent reason. After installing iotop i figured out that the responsible process is tumlerd. Further i have found a year old bug that describes this behavior after deleting a .jpg for example.

Do developers now use SSDs and cant see that they program inefficient code because on SSDs random access is fast? If yes then is it only a matter of time when our software becomes far more inefficient than before the SSDs era and the advantages of SSDs will be not that significant as they were at the beginning?


If you are not using ZFS, you might want to try BFQ:

http://algo.ing.unimo.it/people/paolo/disk_sched/
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Sun Oct 13, 2013 5:07 pm    Post subject: Reply with quote

Thanks ryao.

I was going to ask a different question, but I can't find the source material. Maybe I can find it later.

Regarding performance of ZFS, and recognizing that it wasn't originally designed with performance (speed) in mind, is there an inherent design about ZFS that would prevent significant performance improvements?
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
e3k
Apprentice
Apprentice


Joined: 01 Oct 2007
Posts: 200
Location: Slovakia

PostPosted: Mon Oct 14, 2013 3:11 pm    Post subject: Reply with quote

ryao wrote:
I am involved with ZFS development...

thanks for that. yes i could change from ext4 to something else but it still would be just a workaround (like buying an SSD).
probably it will end like this but this will mean also an end to my hobby: to install linux on old pcs/laptops to prolong their lifetime.
_________________
---__(o)__---
  | |
Back to top
View user's profile Send private message
ryao
Developer
Developer


Joined: 27 Feb 2012
Posts: 123

PostPosted: Mon Oct 21, 2013 3:09 am    Post subject: Reply with quote

pjp wrote:
Thanks ryao.

I was going to ask a different question, but I can't find the source material. Maybe I can find it later.

Regarding performance of ZFS, and recognizing that it wasn't originally designed with performance (speed) in mind, is there an inherent design about ZFS that would prevent significant performance improvements?


I do not see any reason why ZFS performance cannot improve, although it really depends on the metric. If you mean the Phoronix benchmarks, there really is little point to optimize for them. Those rarely test workloads that matter to people.

Anyway, ZFS was designed with both performance and data integrity in mind. However, data integrity is a higher priority than performance. If the two conflict in a design decision, data integrity will win.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Mon Oct 21, 2013 4:05 am    Post subject: Reply with quote

Thanks, just a curiosity. I've read for a long time various comments (not just random people, generally from the Sun blogs) about how it isn't the fs of choice if performance is a top priority. For example, I came across this not long ago, so it makes for a recent example:
link wrote:
I often have to remind people that ZFS wasn't designed for performance. It's fairly clear from the documentation, the initial communication from Sun team, and the source, that ZFS was designed with data integrity as the primary goal, followed I'd say by ease of administration and simplification of traditionally annoying storage stuff (like adding drives, etc) -- /performance/ was a distant second or third or even fourth priority. Things like ARC and the fact that ZFS is just newer than many really ancient filesystems gives people this mistaken impression that it's a speed demon -- it isn't. It never will be.

If your use-case is mostly-read and reasonably cacheable, ARC/L2ARC utilization can make a ZFS filesystem outperform alternatives, but it's doing so by way of the caching layer, not because the underlying IOPS potential is higher (that's rarely the case). If your use-case isn't that, then the only reason you'd go ZFS is for the data integrity first and foremost, and also possibly for the features (snapshots, cloning, compression, gigantic potential namespace, etc); not because you couldn't find a better performing alternative.

_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 4010
Location: USA

PostPosted: Wed Oct 23, 2013 5:28 pm    Post subject: Reply with quote

One thing I wonder...

A lot of these filesystem operations, application software should be dealing with the OS as usual. One thing that I wonder, how many applications call sync() or other OS primitives that force the OS to flush caches? I'd say this tends to be bad programming style, if there's a need to have access to the disk directly, that's really is an OS job.

Also I suspect some OS calls are more expensive than others, and constantly calling them instead of a user-initiated refresh (or a dbus/inotify event) is also bad style... Technically these should be about equal on HDD or SSD if everything is cached correctly, however.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Wed Oct 23, 2013 6:12 pm    Post subject: Reply with quote

On a loosely related note, isn't it FusionIO that was bypassing the OS to allow direct disk access to improve performance? If not them, then it was another storage based start up. I thought it was an interesting approach.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
eccerr0r
Advocate
Advocate


Joined: 01 Jul 2004
Posts: 4010
Location: USA

PostPosted: Wed Oct 23, 2013 8:43 pm    Post subject: Reply with quote

Yes there are sometimes reasons to bypass the OS but this stuff should be part of the OS, not part of userland.
These programs are no longer user applications, but rather servers. And one would hope no more than one of these pieces of software is running at a time, who knows what chaos would ensue if all user applications did their own way bypassing the OS.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed to be advocating?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Thu Oct 24, 2013 1:26 am    Post subject: Reply with quote

Certainly. But for a storage server, the idea seems useful. Perhaps not easily done.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Off the Wall All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum