Don't get lost out in the mountains!

Hey guys, so I do a lot of MikroTik network audits and troubleshooting and I found that there’s a few things about MikroTik that everyone should know, but all the time I’m finding someone who didn’t know that. So, I’m making a blog about some of the things everyone who uses a MikroTik should be aware of, but might not be.

Let’s get to it!

1. SYSTEM PACKAGE UPDATE vs SYSTEM ROUTERBOARD UPGRADE

Many people believe that there’s just one update needed on a MikroTik RouterBOARD, and that’s found by going to System –> Package –> Update –> Check For Updates. 

Unfortunately, there’s also another location that needs to be updated when upgrading the router version, and that’s in System –> RouterBOARD –> Upgrade. 

So you might be wondering, will the router work if you forget to take this additional step? And the answer is: it depends on what you call “work”. I’ve seen everything from unexplained WiFi drops to weird traffic interruption to protocols that don’t behave like they should. For me, that’s broken and particularly so in networks that include hundreds of MikroTik routers that I sometimes find have varying levels of system packages AND differing firmware levels for the router.

Your mileage may vary, but in my experience smooth network operation starts with solid firmware and system packages. Please make a habit of keeping your routers up to date, both for security patches as well as stability and features.

2. SAFE MODE

Safe mode can be both a blessing and a curse.

First, let me explain that safe mode allows you to make changes to a router during a session, and if you get disconnected from the router – meaning that your session gets terminated, for whatever reason – the MikroTik will automatically roll back any and all changes that were made since safe mode was invoked. There’s a safe mode for WinBox found at the top of the window and also for the CLI (Command Line Interface) by hitting Control+X.

[admin@MikroTik] ip route>[CTRL]+[X]
[Safe Mode taken]
[admin@MikroTik] ip route

What’s useful about safe mode is that you can feel pretty confident when doing potentially dangerous activity, such as adjusting/adding/deleting firewall rules or address lists, making changes to routing protocol conigurations, or really just about any change you might be making (including running a speed test that overloads the router or its available bandwidth).

If the user session (WinBox or CLI) gets interrupted at all while safe mode is on, all config changes will be reverted and the router goes back to the last known good state – assumed to be working when you connected to the session that safe mode was started on – since ya’ know, you were able to connect to that session.

***note*** it may take up to 10m for changes to rollback, and up to 100 changes are tracked

That sure sounds like it would be a boon, right? Here’s how it can be a curse – imagine you get involved in customer affecting outage and after some hours, you finally figured out the correct configuration update that got the network back online and into normal service. Awesome and good for you! You wrap up your remote session and shut down your PC/laptop… but FORGOT to turn off SAFE-MODE!!! Oh no… all the changes you just made get undone because the MikroTik router thinks it lost connection to your session and thus, all your changes must be unsafe. I speak from experience here, it’s pretty painful.

By far, it’s worse to have to go on-site and replace a router that you botched up due to NOT using safe mode and making a mistake, so safe-mode is highly recommended. Just remember and get in the habit of turning safe mode off before you call your session done and log off.

Sometimes, if I have multiple changes to make in phases, I’ll turn safe mode on, make several changes, wait a minute to ensure that my session stays connected, then I’ll turn safe mode off for just a few seconds followed by turning it back on again to set a new SAFE configuration save point to rollback to if my next set of commands causes any trouble. There can only be ONE save point, unfortunately.

You should definitely use safe mode, but you should also know how to use it without losing your work.

3. ALLOWING DNS RESOLUTION

I see it less now than a decade ago, but this one remains a concern and still occurs pretty often so it’s still worth mentioning to new users of MikroTik – if you ALLOW-DNS-REQUESTS under IP–>DNS, your MikroTik router can be used for a DNS amplification attack, especially if you have a publicly routable IP. The concern is that on your LAN (Local Area Network) you typically WANT to cache and resolve DNS for your network for best performance, but for the rest of the Internet and on your WAN (Wide Area Network or upstream or Internet), you typically DON’T WANT to answer DNS.

The way an amplification attack works is simple – a bad guy sends a spoofed query for some DNS info, causing the server (or resolver) to look up the info and send back the answer. The problem is that the query part is an extremely small packet and the response (answer) is much, much larger. This allows a small amount of spoofed attack packets to result in crushing a network and sending many multiples of traffic to the target, often resulting in an unusable network.

If you’d like to learn more, here’s a great article from CloudFlare: https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/ 

A couple of ways to circumvent this:

1. Turn off DNS resolution if this router isn’t the gateway+DHCP for the network and it isn’t needed
2. Add a firewall rule that blocks DNS queries from the WAN interface(s)
3. Add a firewall rule that allows DNS from trusted subnets and blocks from all else

Make sure your routers are protected from being used as part of a DDOS (Distributed Denial of Service) attack. I’ve seen customer routers get blacklisted for being part of a bad actor subnet and also congestion on networks where Tiks were being used for the DNS amplification.

MikroTik has some additional tips if you want to learn more about securing your router: https://wiki.mikrotik.com/wiki/Manual:Securing_Your_Router 

4. SOMETHING TO KNOW ABOUT PING

I’ve run into this one more times than I can count. The network isn’t working and you’re testing a continuous ping out to cloudflare (1.1.1.1) or google (such as 8.8.8.8) with nothing but timeouts, and then you’re pretty sure you fixed the configuration, but pings continue to timeout. Then, you hit STOP and START and pings are suddenly working again! What’s up with that???

Well, it turns out that in the routers firewall facility, there’s a table known as connection tracking, or CONN-TRACK. You can think of it as a session tracker for each connection in, out and through the router, which I call a flow of data.

When you fire up a ping, a socket gets built on the router and conn-track gets filled in with a tuple, or row, of all kinds of details about that flow. There’s data about the source and destination, the bytes and packets, state of the connection and the ports and protocols involved all get tracked here.

***As a point of interest, all NAT happens here, so if you want to see what public ports are bound to what private ports and you’re using NAT, this table is where to look under IP –> FIREWALL –> CONNECTIONS

So, sometimes the ping flow gets initiated during the broken state of traffic and the conn-track table details are filled in for the broken flow. In order to get a good ping through, you need to establish a new flow (or session) so that conn-track doesn’t use the old (broken) flow. And that’s exactly what happens when you hit stop and start on the ping tool.

It’s a good idea to get in the habit of using START and STOP buttons when using /tool PING to validate configuration updates.

5. ROMON

One of the coolest things you can do with WinBox is use ROMON to manage a connected network of MikroTik routers. ROMON stands for Router Management Overlay Network, and once configured it allows for an administrator to manage a slew of routers that live on the other side of a MikroTik router or firewall.

For example, if I have a building with 50 MikroTik routers/switches/APs, where all of them are NAT-ted behind the first one, but they’re all physically connected to each other, I can use ROMON to proxy WinBox to any one the routers and then I’d be able to access all the rest of them as neighbors. You just have to turn on the protocol and then tell it which interfaces to run on (and which to block).

Things to know about ROMON:

• It isn’t enabled by default – you have to add configuration for it if you want to use it
• The SECRET option is NOT for encryption of the traffic, just for authenticating neighbors
• It runs seperately from L2/L3 configuration, so you can reach routers that are physically connected even if switching/routing is borked
• It supports ping (to test connectivity), SSH for remote CLI access, and WinBox

If you’d like a more comprehensive look into ROMON, Rick Frey calls it “Possibly the most valuable tool” on his blog: https://rickfreyconsulting.com/romon-possibly-the-most-valuable-tool-in-your-network/ 

Thanks for reading!

If you’re in need of centralized MikroTik fleet management and automation, check out RemoteWinbox or for on premise solutions there’s RWB Admiral, and Thaea Custom MikroTik development