Flash drives, over time, are not as good as new. Especially when working on near full capacity. The more you push the drive the more it shows its age. Endurance cards have additional "reserve/spare" cells they resort to using to allow them to hide their age better. Leaving unpartitioned space on any drive has similar effect as it leaves "spares" to be used as needed.
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Flash drives, over time, are not as good as new. Especially when working on near full capacity. The more you push the drive the more it shows its age. Endurance cards have additional "reserve/spare" cells they resort to using to allow them to hide their age better. Leaving unpartitioned space on any drive has similar effect as it leaves "spares" to be used as needed.
A few points for you to consider.
While the graphs I shared are their "heaviest" test, you could have went two pages further to the lightest test page. Pick one of the bigger "offenders" - the Intel 660p. Keep in mind all the drives on those graphs are with extremely superior controllers to what is on a $30 USB flash drive. The Intel 660p's read and write speed are 1,101MB/s / 208MB/s on a continuous bases. You will not see that in a normal USB drive. Do not overlook that when talking us, ms and seconds and wheter it proves your point, it does not. The Intel 660p light test - avg. latency 80/996us (empty/full) and 99 percentile is 761/18649us (empty/full). So there you have it. While I do not have data to back it up, my educated guess is the avg. USB drive can easily have 1/4s (or higher) 99 percentile write latency.
Another important thing to not overlook is that we do not know how Tesla writes to the disk. Not in terms of sequential/random but rather the IO write request size. If I am writing the software that need to put the video on the disk, and consider the (maybe the most) important scenario - a severe crash with possible auto battery power cut off, I would try to buffer minimally and flush any buffers immediately so as to not lose the most important - the moments before the event. That is to say, the data stream from 4 cameras may cause multiple IOPS and at small request size. If someone has a drive that is failing the Tesla needs, I suggest using one of the oldest and simplest free disk benchmark tools - ATTO disk benchmark (see Download ATTO Disk Benchmark 4.01.0f1). Check the boxes "direct IO" and "bypass write cache", set queue depth at 4, set IO size 512 - 4k and share the 4 readings for disk write speeds. If someone wants to go a bit deeper, I recommend the "gold standard" (at first looks complicated but it is not) which I use all the time at work - TechNet DiskSpd: A Robust Storage Performance Tool
Lastly, your comment that the issue is with different drives and camera versions and firmware. I often see people making the comment that they would reformat the drive and the issue would go away but a few days/week later the issue is back. That if anything proves it is the drives failing. Also, as I have said before, the same Name of a drive is not necessarily the same hardware. Manufacturers often will keep the name and physical appearance of a USB drive but change the internals to suite them better -a new cheaper supplier, a new cheaper controller, shortage of specific NAND size chips, etc, etc. Also unless two drives are same age and had the same workload on them in the past they will perform differently.
Knightshade
Well-Known Member
Even if you did have data to back that up, that'd still be 20 times lower than that other guy was suggesting for drive latency...hence why I mentioned your data debunks his claim.
Here we get into "you can't have it both ways" though
Not sure how? If the issue as the other guy suggests is just some "when a perfect storm of events happens for just a moment the drives specs are too slow just in that moment" then taking it out, reformatting it, won't magically make it fast
But if it gave the too slow due to poorly written software that had an issue just in that one moment, doing nearly ANYTHING would make the message go away for days/weeks until the bad SW glitched again.
And we ALSO have folks who say they just pull the drive, wait 5 seconds, then put it back in and it works for days/weeks again. No reformatting.
Which would be exactly what I describe- not bad HW. Bad SW.
So would drives (across types and brands) that worked fine before a SW update, but suddenly intermittently gave the "too slow" message after it. Which again is exactly that we saw in later versions of V9.
True- though given the dashcam has only existed for ~1 year, and we know the amounts of data it writes, even someone running the system 24/7/365 wouldn't have used even half the write cycles yet on even the cheapest 128GB storage devices. Probably not even on a 64 given the fewer cameras ran for much of the time the feature existed, and most folks don't run 24/7/365 especially since for a chunk of time Sentry was never on unless you explicitly turned it on and most owners wouldn't have at home.
No, all flash storage is only "good" for some specific number of write cycles...(how many can vary depending on the type of flash used, but it's ~1000 cycles on the low end consumer stuff... 3-5k on the "decent/good" stuff... and 5-10k on endurance stuff... (and higher on crazy expensive Enterprise grade flash).
The main thing there though is Tesla is only 720p cameras at ~30fps, so they go through storage a lot slower than a similar set of 4k dashcams will. One 'write cycle" is, roughly, "the size of the drive" so it has to write, say, 128GB total, to "use" 1 cycle.
There's a number of posts with more detailed math if you want to see it in the thread.
To answer some of the comments.
1. Noone tests USB drives for latency as this is not a factor for their typical usage. You can do a test yourself of latency, rather simply when I think of it. First turn off write caching on the drive. Open task manager (if you use Win 10) and go to the disk performance tab. It has an average latency metric. Start 4 simultaneous copy operations of small files onto the drive and for good measure start a simultaneous disk benchmark with something like ATTO on the drive. No this is not "cheating" by asking the drive to do more than what it does for the DashCam copy. We are just trying to get the average latency (what is visible in task manager) to be the 99 percentile latency.
2. Formatting the drive leaves it with blank allocation tables (in case of FAT(32) file system) and all its space free. Both contribute to less need for housekeeping (on the OS and controller level) and reduce the number of operations needed to write the video.
3. Write cycles is not the only way a drive ages. The individual cells, especially on multi-bit cell drives (TLC needs to be able to accurately tell apart 8 different voltage levels in each cell as it needs to store 3 bits in each cell), do age and are more likely to produce a reading that is inaccurate (not what was written). Most flash media has some sort of error checking and correction. This is often done and is invisible to the user or the computer but takes additional processing and slows things down increasing latency.
4. Further on your calculation of how many cell cycles a minute of video takes is not taking into account things such as write amplification, overprovisioning and other factors. We do not know how Tesla writes to the drives. Say you need to write 1KB to the disk but it uses a 8KB page. The 1KB write will have the same affect of writing 8KB, as flash is erased by whole blocks (pages). Nor do you correctly point to 1000-10000 cell cycles as the correct one for nowadays USB drives. Take a look at say SSD Product Warranty | Support | Samsung V-NAND SSD and other similar warranty places. The warranty (yes, some drives will outlive their warranty) is usually given in years but more importantly for our discussions in TBW or DWPD. Reverse calculate from those numbers using the capacity of the drive and you get 300 (maybe up to 1000 cycles on the high end reserved for highly overprovisioned drives). Even the bottom end of 300 cell writes is for SSD drives which, I am sure, noone will argue are still better than the majority of USB drives. So the number of full overwrites of a USB drive for can be expected to perform flawlessly is less than 300. Here are two more NAND reselers telling us TLC is good for 300 - 500 cycles Solid State Storage Solutions, Flash Memory Products - Cactus Tech and Flash Device Endurance: Effect of Flash Type, Controller Type & Workload . Maybe some of it is just taunting the advantages of using "industrial" grade flash, but maybe there is some truth in that.
1. Noone tests USB drives for latency as this is not a factor for their typical usage. You can do a test yourself of latency, rather simply when I think of it. First turn off write caching on the drive. Open task manager (if you use Win 10) and go to the disk performance tab. It has an average latency metric. Start 4 simultaneous copy operations of small files onto the drive and for good measure start a simultaneous disk benchmark with something like ATTO on the drive. No this is not "cheating" by asking the drive to do more than what it does for the DashCam copy. We are just trying to get the average latency (what is visible in task manager) to be the 99 percentile latency.
2. Formatting the drive leaves it with blank allocation tables (in case of FAT(32) file system) and all its space free. Both contribute to less need for housekeeping (on the OS and controller level) and reduce the number of operations needed to write the video.
3. Write cycles is not the only way a drive ages. The individual cells, especially on multi-bit cell drives (TLC needs to be able to accurately tell apart 8 different voltage levels in each cell as it needs to store 3 bits in each cell), do age and are more likely to produce a reading that is inaccurate (not what was written). Most flash media has some sort of error checking and correction. This is often done and is invisible to the user or the computer but takes additional processing and slows things down increasing latency.
4. Further on your calculation of how many cell cycles a minute of video takes is not taking into account things such as write amplification, overprovisioning and other factors. We do not know how Tesla writes to the drives. Say you need to write 1KB to the disk but it uses a 8KB page. The 1KB write will have the same affect of writing 8KB, as flash is erased by whole blocks (pages). Nor do you correctly point to 1000-10000 cell cycles as the correct one for nowadays USB drives. Take a look at say SSD Product Warranty | Support | Samsung V-NAND SSD and other similar warranty places. The warranty (yes, some drives will outlive their warranty) is usually given in years but more importantly for our discussions in TBW or DWPD. Reverse calculate from those numbers using the capacity of the drive and you get 300 (maybe up to 1000 cycles on the high end reserved for highly overprovisioned drives). Even the bottom end of 300 cell writes is for SSD drives which, I am sure, noone will argue are still better than the majority of USB drives. So the number of full overwrites of a USB drive for can be expected to perform flawlessly is less than 300. Here are two more NAND reselers telling us TLC is good for 300 - 500 cycles Solid State Storage Solutions, Flash Memory Products - Cactus Tech and Flash Device Endurance: Effect of Flash Type, Controller Type & Workload . Maybe some of it is just taunting the advantages of using "industrial" grade flash, but maybe there is some truth in that.
Knightshade
Well-Known Member
Which was exactly my point- and all the data you provided supports that, and debunks the dude who was talking about multi-full-second-long latency on flash drives.
Yes, but they're writing 30MB files, not 1k files.
When an SSD is writing large amounts of data sequentially (which is what is happening in the Tesla), the write amplification is equal to one meaning there is no write amplification.
The reason is as the data is written, the entire block is filled sequentially with data related to the same file. If the OS determines that file is to be replaced or deleted, the entire block can be marked as invalid, and there is no need to read parts of it to garbage collect and rewrite into another block.
It will need only to be erased, which is much easier and faster than the read–erase–modify–write process needed for randomly written data going through garbage collection.
Now in theory depending on block size you might be using the "last" block of the 30MB file for less than a full block of data... but that's gonna be a rounding error worth of write amplification, not a "flash drive died in 1-2 weeks" situation as a previous poster tried to claim.
1200 actually.
Samsung SSD 970 PRO | Samsung V-NAND Consumer SSD
That's a Samsung SSD with a rated TBW of 1200 TBW. For a 1TB drive.
Which means 1200 cycles.
Finding TBW for usb keys is harder- but here's one MFG that offers them on some flash drives
https://cdn.sos.sk/productdata/b9/76/0d799c5c/apha008gak0eg-3tm.pdf
For SLC, which is the actual high-end stuff I was talking about, they rate their 32GB flash drive at 352TBW.. which is roughly 11,000 write cycles of the full drive. Actually slightly higher than the 10k I suggested.
For MLC it's 29 TBW for the same 32GB flash drive, nearer the 1000 cycles I mentioned.
Jctin
Member
I have this set up
https://www.amazon.ca/gp/product/B07NY23WBG/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
Surprisingly this cheap reader works
https://www.amazon.ca/gp/product/B0028R3NCU/ref=ppx_yo_dt_b_asin_title_o08_s00?ie=UTF8&psc=1
https://www.amazon.ca/gp/product/B07NY23WBG/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
Surprisingly this cheap reader works
https://www.amazon.ca/gp/product/B0028R3NCU/ref=ppx_yo_dt_b_asin_title_o08_s00?ie=UTF8&psc=1
Rothgarr
Active Member
FYI i have been using this since late June and it's already no longer working with Sentry Mode:
https://www.amazon.com/gp/product/B07D7Q41PM/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
The Tesla says the write speeds are too slow. Tried reformatting a few times to no avail.
https://www.amazon.com/gp/product/B07D7Q41PM/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
The Tesla says the write speeds are too slow. Tried reformatting a few times to no avail.
Knightshade
Well-Known Member
same thing I've used for over a year-though the 128GB version- no too slow messages.
As I suggested to the one other guy in this thread who got a defective drive- run a benchmark on a PC and see your results.
Write speeds for sustained writes should be roughly 30-60 MB/s (versus the 4MB/s Tesla requires).
He was getting like 0.1 at one point. Bad drive- return under warranty.
Rothgarr
Active Member
Rothgarr
Active Member
Knightshade
Well-Known Member
Yeah...that looks like the drive is fine, and that's WAY WAY faster than needed for Teslacam.
You're getting 18 MB/s random (generally the worst score for a drive) and 58 MB/s sequential.
Teslas listed requirements are...4 MB/s. And the actual amount of (sequential BTW) data they're writing is only 2 MB/s.
Further evidence the "drive too slow" thing is some kinda random software bug and not, at all, a hardware issue.
I realize that distinction is not SUPER helpful to you right now if something that objectively should work isn't working- so I guess first thing I'd say is check what SW version you're on... if it's older then 36.2 see if you can get the latest update (it's rollout out now in pretty large #s).
No word on if this actually improves the issue or not, but it's the easiest thing to try first, and some previous dashcam SW issues have improved with later updates.
You're getting 18 MB/s random (generally the worst score for a drive) and 58 MB/s sequential.
Teslas listed requirements are...4 MB/s. And the actual amount of (sequential BTW) data they're writing is only 2 MB/s.
Further evidence the "drive too slow" thing is some kinda random software bug and not, at all, a hardware issue.
I realize that distinction is not SUPER helpful to you right now if something that objectively should work isn't working- so I guess first thing I'd say is check what SW version you're on... if it's older then 36.2 see if you can get the latest update (it's rollout out now in pretty large #s).
No word on if this actually improves the issue or not, but it's the easiest thing to try first, and some previous dashcam SW issues have improved with later updates.
Rothgarr
Active Member
Thanks. Odd that it would just start doing that out of the blue. It started last week sometime and did it EVERY time I drove (even after reformatting). Weird for a bug to just start for no reason with no update after it's been working with with this software version (and previous versions).
But I did just update the car this morning to 36.2 so hopefully the next time I drive the issue is gone.
* fingers crossed *
What is the weather like in your city now? If it is cold and sentry mode stops working after you parked your car for a while, it could be the temperature that is slowing down your device. I was told this is normal for consumer grade by a professional.
I built a styrofoam case and wrapped with foil paper. It is -7C out now it is working fine.
If your city is cold now too, an easy way to test. Put it in ur fridge for about 30 min. Test it immediately after removal from fridge. If it slows down significantly, it means you may need a thermal a case like I do.
Another thing to keep in mind please your device is a usb 3 and your mac is very likely a usb 3 too (to check this go apple icon -> about this mac -> overview -> system report -> usb. you may need to scroll down a bit to see usb 3 listed) while our teslas are on usb 2. The result you are seeing on your mac is likely the "full speed" of you device while it is "leashed" by tesla hardware. I honestly do not know how a usb 3 device handles a usb 2 hardware.
Last edited:
Hey Guys I just wanted to share a extra bit of info for those who have transitioned to the Extreme Micro SD card and adaptor option.
I used the Sandisk adaptor (mobileMate 3.0), mentioned in this thread and a Samsung High endurance 128GB Micro SD card, worked perfectly until it just stopped working altogether, reformatting everything didn't work, could see the drive fine on my computer, did a return with Amazon for the Samsung card as I assumed it had failed in some way, got a Sandisk 128GB high endurance card (which was much cheaper) and still it did not work...you guessed it, it was the Adaptor that had failed (I got this suggestion from someone on Tesla's forums) I got the new adaptor today and everything is working again, so the adaptor can fail! I'm not sure how, and it reads fine on the computer but putting it into the car it just doesn't read it....meaning the TeslaCam icon is not coming up.
Also there maybe some confusion about Sentrymode and TeslaCam - Sentry mode can be on but it doesn't mean your drive is working footage is being recorded!!! (VERY IMPORTANT) if the drive is messed up you can see this by the TeslaCam icon, if it's not visible or you got a X and exclamation mark next to it, its not working...either too slow or just failed (as in my case)
When my adaptor failed the TeslaCam icon did not come up at all, SentryMode was on but nothing was recording...Ironically I reverted back to my old Samsung Bar Plus, which like before worked intermittently...which was better than nothing.
I do an update if anything changes on this new setup.
I used the Sandisk adaptor (mobileMate 3.0), mentioned in this thread and a Samsung High endurance 128GB Micro SD card, worked perfectly until it just stopped working altogether, reformatting everything didn't work, could see the drive fine on my computer, did a return with Amazon for the Samsung card as I assumed it had failed in some way, got a Sandisk 128GB high endurance card (which was much cheaper) and still it did not work...you guessed it, it was the Adaptor that had failed (I got this suggestion from someone on Tesla's forums) I got the new adaptor today and everything is working again, so the adaptor can fail! I'm not sure how, and it reads fine on the computer but putting it into the car it just doesn't read it....meaning the TeslaCam icon is not coming up.
Also there maybe some confusion about Sentrymode and TeslaCam - Sentry mode can be on but it doesn't mean your drive is working footage is being recorded!!! (VERY IMPORTANT) if the drive is messed up you can see this by the TeslaCam icon, if it's not visible or you got a X and exclamation mark next to it, its not working...either too slow or just failed (as in my case)
When my adaptor failed the TeslaCam icon did not come up at all, SentryMode was on but nothing was recording...Ironically I reverted back to my old Samsung Bar Plus, which like before worked intermittently...which was better than nothing.
I do an update if anything changes on this new setup.
Rothgarr
Active Member
It's 32F degrees here, but the car is always parked in a garage that never gets below 55 degrees F.
Thanks for the feedback. How long were you using it till the adapter failed? At the moment I've been using the Rocketek adapter with a Samsung PRO Endurance 128GB card since mid-July. This setup has been solid but it just started getting super cold here. Will see if it lasts. Thankfully the adapter is under $10 if it does indeed fail.
I had same thing SanDisk adapter failed after couple of days, found a Kingston card reader and been fine 3 months in so far ok.
Similar threads
