197 points by walterbell 9 hours ago | 7 comments
jwrallie 7 hours ago
> During device initialization, if the system identifies itself as a user terminal, the initialization script automatically writes 41 SSH public keys into /root/.ssh/authorized_keys. Notably, port 22 on the UTA remains open to the local network at all times.

Forty-one? So who does not have root access to "your" user terminal?

yonatan8070 7 hours ago
You, probably.

On a more serious note, is this any different from ISPs having a remote management system for ISP provided routers? In terms of privacy, if SpaceX didn't have access to the user terminal, they could still just capture your traffic on the sattelite or the ground stations

danieldk 6 hours ago
On a more serious note, is this any different from ISPs having a remote management system for ISP provided routers?

Maybe, but in more and more European countries, ISPs are required to accommodate you hooking up your own router/modem. E.g., I am on fiber and if I want to I can hook up my own router directly to fiber with an SFP+ module (I currently use the ISP-provided media converter, but my own router). Lots of tech users use their own Ubiquiti/OPNsense/OpenWrt routers, so no remote management.

I wonder if this requirement applies to Starlink as well, since they are an ISP.

prajaybasu 5 hours ago
As far as I'm aware, they are only required to allow you to use your own router.

DSL tech is far simpler and it's always a combo unit so I could see a case where you would be allowed to bring your own DSL modem.

But it just doesn't work like that for DOCSIS or GPON where the cable modems or ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP, etc. even if they don't do Wi-Fi (so ISPs don't call them "routers" - except SingTel, which uses "ONR" to distinguish these units because they are in fact routers for IPTV and SIP).

For all of those modems/ONTs, the firmware updates and the configuration for telephony/SIP and PPPoE are controlled by the ISP and also tested to work with their OLT or CMTS so it's just not possible for the ISP to guarantee support for any random modem or ONT.

And to support the advanced configuration required these days for VoIP, IPTV, etc. on the "modem" or "ONT", ISPs basically have a backdoor called TR-069 which is really not too dissimilar to what Starlink has access to with their SSH keys.

Even if you get "true" dumb modems or ONTs which do not do any routing whatsoever, the device on the other side still has full control over your dumb device via the DOCSIS provisioning process or GPON's OMCI. Starlink seems to be using SSH instead of building a whole protocol - because satellite tech is proprietary and doesn't need to work on other hardware.

So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.

At most, I think using your own router would require the EU ISPs to provide bridge mode support, but that's not special to EU. However, the TR-069 backdoor is still active even with bridge mode.

It can be fairly easy to stop TR-069 with a "dumb" ONT (usually SFP) but ISPs can and will notice that. Whether they allow it is up to them.

AnssiH 3 hours ago
> But it just doesn't work like that for DOCSIS or GPON where the cable modems or ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP, etc. even if they don't do Wi-Fi (so ISPs don't call them "routers" - except SingTel, which uses "ONR" to distinguish these units because they are in fact routers for IPTV and SIP).

At least in Finland the norm is that you can use your own DOCSIS modem from any manufacturer, you just tell the ISP your modem's MAC address.

Not for GPON, though.

claudex 5 hours ago
>DSL tech is far simpler and it's always a combo unit so I could see a case where you would be allowed to bring your own DSL modem.

Not really, when you want to increase the bandwith, e.g. with vectoring[1], you need to have all neighbor modems to participate, which prevent free modem choice for the users.

>But it just doesn't work like that for DOCSIS or GPON where the cable modems or "ONT" router combo units these days do much more than just media conversion - SIP, PPPoE, IGMP, etc.

In Belgium, the ONT is just media conversion these days, SIP is done on the provider box, so you can have your own GPON SFP.

>so it's just not possible for the ISP to guarantee support for any random modem or ONT.

The ISP doesn't have to guarantee support to let you use your own hardware. It just have to give you the specs to use it and let you plug the ISP box if you can't configure vlan of dhcpv6 client.

[1]: https://en.wikipedia.org/wiki/VDSL#VDSL2_vectoring

claudex 5 hours ago
Each country regulator in the European Union have to set it's own regulation, but the BEREC (Association of UE telecom regulators) guidelines say that in most case, the free choice of router and modem is what's is required by the EU decisions https://www.berec.europa.eu/sites/default/files/files/docume...
bkor 5 hours ago
> So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.

Ziggo (called UPC in other EU countries) uses DOCSIS. The instructions on how to use your own DOCSIS modem are at the following link (in Dutch): https://www.ziggo.nl/klantenservice/apparaten/wifi-modems/ei...

Edit: it really is using your own modem. It's not about putting it in bridge mode.

donnachangstein 5 hours ago
"Your" modem is netbooting "their" firmware and they have full remote access on a management interface.
kotaKat 2 hours ago
This. I can just provision in the backdoor interface on the modem with a config file anyways and gain access.

Plus depending on model (like Arris modems), I can do things like set the password of the day seed (away from the factory default) to further lock it down and gain management access remotely.

chgs 4 hours ago
Not the case in the U.K., dal or fibre doesn’t matter, domestically you typically have a standard sfp or other converter from anywhere which presents it as Ethernet which you then run pppoe over.

Tr069 allows the isp to remotely configure their equipment which most people are happy with, but if you want to use your own then that’s fine, and obviously unless you enable it the isp won’t configure your router or any other equipment.

Starlink acts far more than a media converter.

fx1994 5 hours ago
Provider will not support my modem that is for sure so if you have any issues you are on your own. I use my own Mikrotik + Zyxel PMG3000 GPON SPF and no issues at all.
donnachangstein 5 hours ago
> I am on fiber and if I want to I can hook up my own router directly to fiber with an SFP+ module

I don't think you quite understand how this works.

The ISP controls whatever the other end of that fiber is plugged into. It doesn't matter if the medium is fiber, or copper, or a piece of string. The ISP always has control of the other side of the customer interface. It doesn't matter if the box physically resides in your home or not.

In the case of Starlink, it's all contained within one box.

In the case of DOCSIS (cable), you may physically own the modem, but the ISP controls the firmware it netboots and has full remote admin to the device.

mattclarkdotnet 5 hours ago
Owning the modem only gives them access to the link layer. It didn’t give them any access to my router or to my https traffic
jmb99 4 hours ago
Do you think root access on your router gives access to your unencrypted https traffic?
lupusreal 1 hour ago
It gives them access to the LAN so they can, for instance, figure out how many internet gadgets your house has and sell that information to advertisers, or do even worse than than.
conorjh 2 hours ago
depends
miki123211 3 hours ago
Starlink is not an ISP in the traditional sense.

If a normal ISP wants to operate in country a, they need infrastructure in country a. This means they either follow country a's laws or that infrastructure gets seized.

Starlink could just as well be operating entirely from the US, and there's very little that foreign governments could do to stop them if they break some foreign laws. They could make payments and shipping complicated, which is probably why Starlink would rather comply if the requests are somewhat reasonable, but Musk has indicated multiple times that he's willing to stand up to unreasonable restrictions if the need is dire enough.

ratatoskrt 3 hours ago
This is not at all how laws actually work. If you sell a product to customers in a specific country, you generally have to comply with local laws. You might be able to avoid this if you're very small and your service is entirely virtual. However, as long as Starlink needs to provide their customers with physical hardware, there are numerous ways to enforce regulations.

And you can always go after people - and I mean both Starlink executives and customers.

dangus 3 hours ago
Right, if this is how it worked we’d have to explain how companies like DeepCool and Huawei are banned from doing business in the US.
zdragnar 4 minutes ago
Half of the EU have also banned ZTE and Huawei equipment, so it's not just a US thing.
verzali 2 hours ago
Some countries require you to have ground infrastructure in the country to operate satellite systems. Starlink's architecture also means they need to have a lot of ground stations all over the place. They can skip some areas if they need to and still offer service, but they'd seriously struggle to provide a global service by using only US ground stations.
conorjh 2 hours ago
"They could make payments and shipping complicated..."

eg they could outright ban the sale of StarLink products, ya know, being in charge of the laws and all

lupusreal 2 hours ago
AFAIK, in America, ISPs are required to permit user-provided modems as long as those modems are technologically compatible. I believe the Television Viewer Protection Act of 2019 did this, although it was already the norm for ISPs to permit third party modems before this.. I guess because they knew if they pushed the issue they'd lose anyway, given the precedent of telephones, cable cards, etc.

The problem then with Starlink is nobody is manufacturing compatible third party Starlink terminals, at least yet.

_joel 59 minutes ago
ronsor 7 hours ago
If they have access to the router, they can capture local network traffic as well as internet traffic.
arghwhat 6 hours ago
Note that such capture would be quite terrible for performance, not only requiring disabling any hardware offload (a great router might be able to route a few hundred megabit in large packets without offload assuming it doesn't do anything else) to make packets visible for capture, but it would also have to stream the output back to the adversary over the uplink as you would be limited to at most a few gigabytes of local, extremely slow storage, giving no means for local offline analysis...

The risk of access to the router is more that they can access your network and touch unprotected and vulnerable things rather than active monitoring.

db48x 1 hour ago
And most of that wouldn’t be useful. We use encryption for almost everything now for a reason.

No, wiretaps on modern networks do not rely on backdoors, or even big labeled front doors like SSH, on individual subscriber devices. Instead it is built into the lower level routing. When an ISP gets a warrant (or whatever relevant document your country uses) they configure their routers to tag all of your traffic and mirror it to a server to be recorded. It’s entirely invisible to the subscriber, and highly automated.

edf13 5 hours ago
I sit my external routers (and Starlink) behind another network (DMZ).
amelius 4 hours ago
I think the main problem is that they would have access to other traffic on your local network. But you can just add more hardware to isolate it.
znep 6 hours ago
I wonder who would be best equipped to see if any of those keys are traceable to individuals involved in special government affairs lately? There have been some good leaks...
niobe 6 hours ago
could simply be 41 instances of the same server in 41 regions, not necessarily a cause for concern. Starlink is a global service after all. I'd be more concerned if 41 instances were sharing one key.
overfeed 5 hours ago
> I'd be more concerned if 41 instances were sharing one key.

Dozens/hundreds/thousands of web servers servers can easily share one private key in a certificate, public keys offer even more options on sane designs. Directly authenticating 41 servers using ssh-keys is just poor, slap dash engineering.

jmb99 4 hours ago
I would argue reusing private keys worldwide is slapdash engineering. You generally want to minimize exposure in the event of a breach, not maximize it.
petee 47 minutes ago
I wonder what the reasonable balance between reuse and over exposure is; I'd think you would want less keys per device, and have less key overlap (ie more keys overall.) But forty two sounds high, and isn't it now just 42x more at risk?
overfeed 3 hours ago
> I would argue reusing private keys worldwide is slapdash engineering

I wasn't suggesting it, and frankly can't see how that could be a solution in this instance. I was making a comparison against current practices on a harder problem to solve , i.e. safely scaling a single private key in an SSL certificate across many servers is solved today without a 1:1 server to certification ratio

gardenerik 4 hours ago
Is it a better idea to share private keys? In case of server breach, you will have a much harder time, won't you?
ta1243 2 hours ago
A better idea would be the terminal trusting one or two core certificate authorities and then those authorities creating time limited certificates when needed.

So the terminal accepts "sshauthority1"

Then the 41 remote sites contact sshauthority1 to get a 1 hour (10 minutes, 10 days, whatever) long certificate for "site18"

If a remote site is compromised sshauthority1 no longer issues certificates, and within an hour (10 minutes, 10 days, etc) the remote site can no longer reach the terminals.

Revoking a key from that many terminals (many of which will be offline) if one of the 41 keys is exposed is not trivial.

Now if sshauthority1 is compromised then you've got the same issue with rotation (although can CRL it), but it's easier to secure one or two authorities than 41 keys.

overfeed 3 hours ago
> Is it a better idea to share private keys

It is not, amd I can't see how my earlier comment can be read as recommending that. This is a solved problem for private keys (using load balancers, for example) , so public keys are lower-hanging fruit than that.

Edit: upon rereading, I cam see how the word "share" would be ambiguous in the context of if a private key. I meant "jointly make use of", rather than "distribute copies throughout the fleet". I have exited my root comment to make my meaning clearer.

MrOwen 5 hours ago
Is that normal? I would imagine that if I were managing such a large deployment, I would just use a CA for the keys and then issue CA signed private keys so that I don't need to add a bunch of random ones to authorized_keys
steveharman 3 hours ago
Surely anyone with any imagination inside Starlink would have insisted on there being _42_ keys?

Perhaps Elon _doesn't_ have a brain the size of a planet after all.

ronsor 7 hours ago
You, of course.
adolph 52 minutes ago
> So who does not have root access to "your" user terminal?

Not to defend this but curious: Unless the terminal is attached to a local network that also has internet access, the satellite network would need to be traversed in order to connect using those keys, right? What kind of NAT/etc does Starlink use for satellite routers?

84634E1A607A 21 minutes ago
/56 DHCPv6 Prefix Delegation, and NAT44444 for IPv4 (192.168.1.x/24 for router, 192.168.100.x/24 for dish, 100.64.x.x for CGNAT, 172.16.x.x for Ground Station inner net, finally you are on the Internet).
e2le 4 hours ago
It would've being interesting if they had published those public keys.
sneak 4 hours ago
I’m a single user and my authorized_keys is 25 lines. I have different yubikeys in laptops, keys on iPads and iPhones, and secure enclave keys on macs.

I imagine starlink has more than 1-2 sysadmins. I think a hundred pubkeys would be reasonable.

tomalpha 2 hours ago
It seems wrong that each individual sysadmin human in Space X would need to (a) login to my device remotely, and (b) require individual credentials to do so.

Having some way to remotely push updates, and having some kinda of (preferably with your consent!) remote access might be reasonable, but I would expect that to be via some kind of intermediate gateway/app/something and not direct from a sysadmin’s individual account.

ta1243 2 hours ago
SSH Certificates is a far better approach even if that many users needed direct access. It's not 1990 any more.
latchkey 8 hours ago
Discussions on similar submissions:

Teardown of the SpaceX Starlink User Terminal https://news.ycombinator.com/item?id=25277171 (December 2, 2020 — 158 points, 138 comments)

nine_k 6 hours ago
Dear author, please consider fixing the typo in the title, it currently reads "Ternimal".
philipwhiuk 3 hours ago
Classic keming issue.
purpleidea 3 hours ago
Post the 41 public keys, we can see which devs use them probably.
walterbell 9 hours ago
breppp 4 hours ago
> DARKNAVY built a basic QEMU-based emulation environment for the Rev3 firmware

Anyone has links to resources about how to emulate a firmware that connects to external devices (GPS here), any ready solutions?

walterbell 1 hour ago
https://android.googlesource.com/platform/external/qemu/+/2d...

> Android Emulator is downstream from the QEMU emulator. It adds support for booting Android devices, emulates typical Android hardware (OpenGL, GPS, GSM, Sensors) and a GUI interface. The android emulator extends qemu in various ways.

londons_explore 7 hours ago
I'm surprised to hear all packets are processed in userspace...

If one is doing 1Gbps of traffic which is 100 byte UDP packets, that's a million packets per second you're gonna need to process.

A 1Ghz CPU only then gets 1000 cycles to process each one...

Very doable, but certainly not easy unless your engineers like hand coding assembly and having to think about every lookup table trick in the book...

hackernudes 7 hours ago
> Drawing on existing research [3], our preliminary analysis of these programs and configurations suggests that the network stack architecture is somewhat similar to DPDK [4], mainly relying on a user-space C++ program to bypass the kernel for handling network packets.

The way it usually works is that the initial packets are handled in software but once the endpoints are established it flows through hardware. Sometimes certain patterns are always handled in software. The software could be a patched kernel or a XDP style kernel bypass.

Source: worked peripherally on an Intel Puma cable modem router/gateway that used DPDK or something like it. So I'm not 100% sure, but it is an educated guess.

dilyevsky 4 hours ago
> I'm surprised to hear all packets are processed in userspace...

Specifically for cases of forwarding DPDK-style approach can be faster because it will incur fewer buffer copies.

Starlink only does 25-200Mbps and average packets are like 7-8x larger so at most you're doing ~36000 PPS which is pretty manageable even on 1Ghz

_joel 56 minutes ago
Doing it in userspace aviods another memcpy, it's much faster.
riehwvfbk 7 hours ago
Why would it be any less efficient than processing the packets in the kernel? There's a way to map the hardware queues into userspace (the article talks about the system being DPDK-like). At that point why does it matter that the polling code isn't in the kernel?
londons_explore 6 hours ago
Most hardware >100Mbps has hardware offload - ie. the hardware is told which packets to send where, and software doesn't touch individual packets (except rare packets like ping).
6 hours ago
rapsey 6 hours ago
> which is 100 byte UDP packets

100 byte?? Starlink has regular 1500 byte MTU.

tuetuopay 4 hours ago
In networking, it is the norm to measure performance in packets per second, so with small packets. Unless you're performing DPI or encryption, routers only use the headers to take routing decisions, so whether the payload is 10 bytes or 1000 bytes does not matter: the processing cost will be identical. Only the hardware bandwidth will matter for large packets, though this is rarely the issue (I've hit DDR4 limits once using XDP, and fixed by adding another stick of memory);
Tepix 4 hours ago
With RTP traffic you often have lots of small packets.