Well, my second post after being registered more than one year.
I found the problem related to this thread http://www.urbanterr...pic/27825-drdos yesterday in the morning when I joined to one of my servers and I noticed some lag. After that, I visited the other servers and had some lag too.
Then I contacted support because I noticed some cpu spikes in my server, and being a shared environment (VPS) probably I had a neighbor abusing the resources. Certainly there was one, and then he was kicked from my node. But the problems still were there.
So, I decided to look at other graphs, and noticed big network spikes around 1 to 2 MB/s. And after that, tried iftop and tcpdump to track the traffic leak. Found the problem with some IPs using lots of traffic in the UrT ports.
I started to block via iptables and report IPs like stupid almost half of the day.
After that, I told to Durandal from the DSG clan about the DoS problem, and he redirected me to the post previously mentioned.
Then, I received some responses from the abuse mailboxes, and one of them pointed me to the problem, we aren't the victims, but we are the tools to attack that IPs.
So, this problem have several consequences:
- High CPU usge: I noticed an increase about 5-10% CPU when the attack was active. This translate in noticeable lag (little or high spikes, depending on how much packets are being reflected), and probably a penalization of your hosting company if you are in a shared environment.
- Bandwidth usage: This other translates in bandwidth waste, and you probably will exceed your monthly quota. Also, can be another cause of lag, and other possible penalization of your provider.
- DDoS: Yes, we are the DDoSers, so we are exposed to a possible report from the attacant IP's.
In conclusion, these are SEVERAL problems for us, and for the other people getting attacked.
I spent the rest of the day researching for a solution to this.
Unfortunately what I know and what I found about iptables was useless. Some people point you to the connlimit module, and others to the limit module, even to the state module in iptables. I was unable to use all of them, because:
- connlimit and state: These are about TCP connections, and UDP is connection-less
- limit: This may work, but the idea is to block a host after sending (or receiving) xxx packets, and this module block you globally after the imposed limit and not per host, so you will DoS yourself when the attacants come.
After reading the article here: http://www.lemuria.o...tion-drdos.html I was thinking on the possibility of limit the packets containing the getinfo and getstatus strings via iptables and his 'string' module. Unfortunately, I am using VPSs in OpenVZ, which doesn't support the vast majority of the uptables modules. So, working on a solution for this is useless for me.
Then, I reviewed the code of the conneclimit.patch file from the previous post, and thought that I can create something similar for the getinfo and getstatus commands.
So, here is my workaround. I checkout the ioquake project from the SVN repository of Rambetter and worked from it.
I am unable to create a .patch because I don't know how to use it properly, and this must be done ASAP :D
Notes: I read the new rules talking about no modifying the code and so, but I think this is necessary, and also lots of people talk about patching and so :P
- Checkout the project from the SVN. Here the thread about it http://www.urbanterr...-exploit-fixes/
- Apply all the patches that you want according to the file
ioquake3-UrT-server-4.1/extra-patches/README-extra-patches.txt
- Find this file
ioquake3-UrT-server-4.1/code/server/sv_main.c
To fix the getinfo command:
- Find this line
void SVC_Info( netadr_t from ) {
- Replace with this this code
int lastTime = 0; void SVC_Info( netadr_t from ) {
- Find this line
char infostring[MAX_INFO_STRING];
- Replace with this this code
char infostring[MAX_INFO_STRING]; int currTime = svs.time; if(currTime < (lastTime + 500) ) { return; }
- Find this line
NET_OutOfBandPrint( NS_SERVER, from, "infoResponse\n%s", infostring );
- Replace with this this code
NET_OutOfBandPrint( NS_SERVER, from, "infoResponse\n%s", infostring); lastTime = svs.time;
Now for the getstatus command (this is even more important).
- In the same file, find this
void SVC_Status( netadr_t from ) {
- Replace with
int lastTime2 = 0; void SVC_Status( netadr_t from ) {
- Find this line
char infostring[MAX_INFO_STRING];
- Replace with
char infostring[MAX_INFO_STRING]; int currTime2 = svs.time; if(currTime2 < (lastTime2 + 500) ){ return; }
- Find this line
NET_OutOfBandPrint( NS_SERVER, from, "statusResponse\n%s\n%s", infostring, status);
- Replace with
NET_OutOfBandPrint( NS_SERVER, from, "statusResponse\n%s\n%s", infostring, status); lastTime2 = svs.time;
And that's all. With this I am reducing the responses of this commands to a max of 2/second.
This can have some implications.
- The server list may be unable to detect your server if someone else did a request in the previous 500 ms.
- Gametracker (or other trackers) and the "Server info" button in the server list may get an empty response in some moment.
At this moment I am still able to see my server in the list and get the server info without problems, despite the packet flooding.
I am still looking for more cons of this method, but I hava found several advantages.
- The attackers are still sending commands at a rate of 5-10KB/s, but we are not wasting lots of bandwidth in the response. Going from 1-2MB/s to just some KB/s. Then we will reduce the probability of getting reported, and we will not waste our precious bandwidth quota.
- The CPU consumption dropped almost to the normals (The server still must handle this commands).
- The lag spikes dissappeared (for me).
Well, that's all. This has been from my experience with this crap, and I think that may help some other server admins.
If you approve this code as a temporary workaround, I think there are some other things to do.
- Find a workaround at the Operative system level. Particularly interested in IPtables, and preferably with the modules that have access any people despite their Virtualization technology if they use.
- Create a .patch file
- Adjust the rate to a possible better values instead of 500 ms.
- Create configurable options via server vars.
- Create a better solution in the code, probably limiting by host and not globally. This can solve the disadvantages that I've exposed.
- Create precompiled binaries for some OSs.
And... the end.
I am not a C coder, or the best programmer. Even I am not an English native speaker, so, sorry for it and if something isn't clear, please tell me to fix it.
I hope this can help you.
I hope you read to the end.
I hope I will not get flamed :D
This post has been edited by ipwnn00bs: 08 December 2011 - 10:22 PM