Guest Post: ‘A Venture Capitalist’s Experience of Startups and Security’

A security-aware and very technically competent venture capitalist within my circle recently shared an epic rant, which with a little light editing and with requisite permission I have edited into this posting. I think it contains an important message – not least it demonstrates that funders are catching up with the fundamentals, and you’ll soon be less able to get away with merely being ‘the next big thing’:

A Venture Capitalist’s Experience of Startups and Security; by AVC

First published at dropsafe.crypticide.com

Which class of company has the absolute worst security in the world?

Answer? A ‘Silicon Valley Startup’!

I am pissed at people who think that by merely having a CS degree and knowing PHP means they have a shred of a clue about security and why it’s important. In 10+ years of VC, seeing 3500 pitches per year, not once have I seen one lousy dollar nor one head allocated in any of those pitches or spreadsheets to have anything to do with the startup’s security, sysadmin or similar – unless it was a security company, of course.

Not once! Shockingly depressing.

If you reading this and are in a startup, make a checklist of these bad practices and see how many you do? Assign 10 points per naughty action, then calculate your naughtiness score. Now take that score to your VCs and tell them that all that money they gave you to develop that super-proprietary, market leading technology is flying out the window and the value of their investment is therefore squat?”

Wanna bet that causes a firestorm ?

Really, it’s so awful it’s driving me batty. Every one of these items is 100% true and most of them I have seen dozens of times:

FX: please add the sounds of a chiaroscuro violin chorus swelling in the background for effect, with a whiny, arrogant, ‘pleading engineer’ tone to most of the following quotes – you know both the tone and the snivelling look too…

  1. Engineers need easy access to the code!” – How many times have you heard this? Have you never heard of encrypted Git hubs?
  2. The members of the Engineering access group are.. everyone” !
  3. We keep it all mounted under /project..”
  4. We embed the full path and servername in the code header” – sure, so the bad guys know where to go!
  5. We need to work from home / a train / a bar / the planet venus” – so everyone has a checked-out copy of everything on every laptop
  6. …including the finance guy…
  7. (followed by) – “I lost my laptop in Prague at the IETF“.
  8. We have a VPN using ‘COMPANYNAME’ for credentials“. Grrrr.
  9. It’s such a pain to have a complex password on the Wireless Access Point that we just will make it ’1234′ or ‘COMPANYNAME‘”.
  10. We use WEP. what’s wrong with that?
  11. We give the wireless password to every joker who ever visits the company, including the cleaning guys and the guy who restocks the Coke machine” – I have seen this personally – “Hey man, they need to check their email on their iPhones too. They are nice guys, we trust them…
  12. Change the password? “Naw.” Buy a second access point for private use and put it on a private internal VLAN and spend 5 min and maybe $49? “Naw. I’m really busy, leave me alone, you are a moron. don’t tell me what to do, I’m a graduate of CMU.
  13. Oh screw it, no password on the access point is fine” – I have seen this too.
  14. Bad guys using your net to grab porn and stash it on your engineering server? I have seen this, and when the server ran out of disk space someone then said “Hey, why are there 900gb of JPGs under an account name we dont have? WOW, that shit is filthy, let me examine every one…
  15. Having your VM instance running a porn server serving up porn for free to the net at large which you didn’t know about? Yep. Oh and it was at Amazon so you were paying for S3 and EC2 cycles to do it. Why did you never check what goes on up there? Answer: “The guys are really good, they just know what’s going on up there“.
  16. Firewall? “We have a DMZ, that’s good enough on a default ipf/iptables. Ok, ok, we will go to Fry’s and get one, plug it in and we are good.
  17. Change the default network address or password? “Naw too busy!” – I have seen dozens of companies with 192.168.1.x with cisco/admin and the outside admin web interface left on – shodan is your friend in this instance.
  18. We need to open up over 9000 ‘special testing’ ports for the ‘special services’ that Fred/Bob/Guido/Aunt Emma are developing and testing. No, they don’t use SSL, yes they need to connect to MySQL from outside too. We are ok because we moved the MySQL socket to port 23456
  19. Hey, man that drop password table to ASCII sure is an odd operation, wonder what they are doing?” – seen this too often to even be even slightly amused
  20. System Administrator? We don’t have budget for one, so one of the guys does it” – or rather, he doesn’t. He’s too busy.
  21. Dealing with logs? Syslog? Splunk? Outsource it one day a week? – “Nah. Too busy.
  22. Sourcefire? Intrusion Detection? Snort? “What’s with all these pig references anyway?
  23. Lock on the door? Nope, really.
  24. Door left ajar 24×7? Right.
  25. Building 3/4 empty, random people walking through? “Sure. Probably a new hire or candidate anyway. I’m too busy to ask him what he’s doing in any event. Everyone looks the same…” – hipster pork pie hat, three-day stubble, peglegs, yellow keds, bowling shirt, warby parkers – “…so if he wears the uniform, he’s in!
  26. We need to open port 5060 (SIP) so our guys can use their soft phones from Defcon, SXSW or on the road. With auto refill on the credit card for the VOIP provider.
  27. VOIP over SSH? Too tough to set up and maintain, skip it. Every time a new guys joins, gening all those keys and all that junk under Windows is such a drag.
  28. VOIP phones on the same VLAN as the code machines and the finance machines? “What’s a VLAN? What’s a trunk port? Isn’t that something on a ship?” – oh Lord, please make it stop.
  29. Find that the carefully designed and separated VLANs are all plugged in to the same router, with that router totally open to route any-to-any as it was never touched from out of the box?
  30. Ask the engineering team if they have read (much less used) the free NSA guidelines on IPv4, Cisco, IPv6, recommended security practices? “The what? The who?” – Kill me now, please.
  31. Consider the cloud code repository with an account name of ‘COMPANY NAME’ and password of ‘engineering’; $16m has been invested in that company. What’s the bet that every last byte of that repo has been exfiltrated?
  32. Same company keeping its financials in the cloud on QuickBooks with an account name of ‘COMPANYNAME’ and ‘cfo1234′ for the password. Plus their banking information was in there too. Oh, and they were getting ready for an IPO, so we had to remove them from QuickBooks in order to pass the investment bank’s security profile – thank god!
  33. Banking transfer procedures not having at least 2 physical steps involved, such as phone call and 2 sigs or fax and 1 sig etc. We stopped a $486k transfer to Romania via a contaminated PDF this way. The PDF was a spearphish from a financial analyst with a report on ‘VC Performance’ addressed to the CFO. Nicely done, phisherman. The report was actually pretty good too!
  34. Stupid banks that fundamentally don’t understand password entropy and require ‘8 characters of which 3 must be numbers and 2 a symbol‘ so the users write them on a sticky note; this kind of education leads to the services they develop being similarly brain dead.

Runner-Up Prize

Needing to copy the data to a ‘secure’ location, as their boss finally woke up to the risk of only having one copy of the data locally, junior woodchuck engineer #1 says:

“Dude, let’s increase the size of the logical units at the same time, while the system is up, this will only take a sec to resize? Incidentally, aren’t my white laces with the red keds just too hip, man?”

“Fine by me, and yes they are stylin!”, says woodchuck #2.

…and thus we can say adios to 3TB of work, none of which of course was backed up.

Luckily the 13 copies on various insecure laptops were able to be laboriously copied back (phew!) and then hand resolved (diffed) to create a blob which had to be hand walked to get it to compile again.

Did they make their release date? I think not.

Woodchucks #1 and #2 now are pulling doubleshot lattes with skim milk and ventes at a place with free wifi so they can work on their next startup idea.

Grand Prize Winner

Sending a crashed SATA drive – no backup of course – containing HR/employee personal information including banking, health care, family, etc, info on employees to the site they googled as being the world’s best disk recovery service – rated #1 by users! Guaranteed satisfaction! Most trusted name in the business! – which in the end turned out to be a mail drop at Mailboxes Etc for a Bulgarian operation of lesser trustworthiness.

Excuse? “But I was in the middle of a release!”

2 thoughts on “Guest Post: ‘A Venture Capitalist’s Experience of Startups and Security’

  1. Dave Walker

    Cool article. Have done some tactical forwarding of the link; will see how some UK start-ups fare, by comparison :-).

    Reply
  2. Francis Davey

    Great article. You could feed back to your VC that my experience working as a systems administrator in a startup is what persuaded me to become a lawyer. I felt then (as now) that management and investors didn’t really appreciate the value of getting these things right.

    A startup can be expanding very fast, putting a lot of demand on a sysadmin who has to build a completely new infrastructure. It is much harder than maintaining the integrity of an existing, properly constructed, system, which is of course hard enough.

    Some of the problems were not security related, but others were. It was hard to convince management that my professional views were just that, professional. That I might know things they did not know and that my judgment calls were informed by experience. For example a system that expires everyone’s password every month, storing *all* old passwords to prevent re-use was perhaps not the wisest idea.

    So, I decided to retrain to a profession where management do seem to assume that you know what you are talking about and if you say “you will go to jail for this” they sit up and take notice. I now advise the same sorts of startups as I once worked for and try to keep them out of legal, rather than technical, trouble most of the time.

    Reply

Leave a Reply