One of our Core Routers had an RSP
failure. Typically our RSP's will fall
over to another RSP as we have dual
RSP's in our routers. However when the
first RSP crashed the second one
failed. We will be replacing the slave
unit immediately in the next week or
two. The result of this failure was a
partial outage and loss of 4.5Mbit/s of
bandwidth, as well as the loss of two
remote POP's. All web hosting,
colocation and dedicated server
customers would have been relatively
unaffected. Our other core router
remained running and connected to the
internet. It should be noted that this
failure was NOT the result of the
recent IOS vulnerability. We have
patched all of our routers as of July
17th, 2003 and have applied filters to
block any such attacks on our remaining
routers. We apologize for any
inconvenience this may have caused.
On Tuesday July 15, 2003 Cisco
Systems announced that "Cisco routers
and switches running Cisco IOS®
software and configured to process
Internet Protocol version 4 (IPv4)
packets are vulnerable to a Denial of
Service (DoS) attack." Although Cisco
Systems states that the vulnerability
is to "a rare sequence of crafted IPv4
packets with specific protocol fields",
Bella Mia Network management has
determined that this vulnerability
constitutes a sufficient security risk
to require immediate action.
Cisco Systems has ascertained that
the new version of the Cisco IOS®
software will resolve the security
issue. Therefore starting tonight and
continuing for the next several days,
Bella Mia Engineers will be upgrading
all Bella Mia IP Network routers to the
latest Cisco IOS® software.
It is Bella Mia's goal to provide
the highest level of service to our
customers. We appreciate your business
and look forward to continuing our
relationship in the future.
For additional information regarding
the Cisco Systems announcement, please
visit their website at:
We have successfully moved two DS1's
over to Fiber. Both T1's are connected
to two of our remote POP's. There
should have been no noticeable down
time as these are backup links.
We have completed moving our DNS
servers over to our new IP space.
We have finally turned up our
4.5Mbit feed from AT&T. You can
view traffic graphs here:
ACC-AT&T
All Upgrades are complete.
We are currently replacing our
RSP2's with RSP4's in our Core 7513
router. While we perform this and other
upgrades to the IOS we will be powering
down this router. This should not have
any ill-affects outside of slight
latency with the loss of our Savvis
connections while we perform this
upgrade.
We should also have our AT&T
Feeds live today as well adding
4.5Mbits of Bandwidth to our Network!
Finally once our RSP4 is in place an
all upgrades completed we will again be
taking full routes from all carriers.
SBC finished installing 3 more T1's
today. These are our AT&T feeds. We
hope to bring these live with AT&T
today with any luck!
We will be provisioning two new T1
Circuits over our Fiber. No outages are
expected.
As of 3:43PM CST our C&W
connections have been restored. We
apologize for any inconvenience.
We are currently experiencing an
outage with our Cable and Wireless
Circuits. We have a ticket into C&W
and expect a resolution shortly.
We have ordered an additional
4.5Mbit feed from ACC Business, a
division of AT&T. ACC currently
peers with UUnet, Level3, Abovenet, and
several other carriers. We feel
that ACC/AT&T's bandwidth,network
reach and peering will be a great
addition to our network and its
strength as we open the new year! We
expect to be up and running with ACC as
of January 20th, 2003.
We have asked ARIN for our own
private and portable IP space. We have
applied on three separate occasions,
each time being turned down. We feel
now, and have felt in the past that we
met and meet all the requirements that
ARIN has set forth. We are multi-homed,
run BGP, and have our own ASN. We have
set up an RWHOIS server to list exactly
what we do with all our IP space
currently and in the future and are
confident that we will get our own
space. Once this happens it will ensure
that our customers are not forced to
move from carrier to carrier as they
either go bankrupt or fail to provider
adequate services. In any event once we
receive our allocation and have been
successfully announcing our blocks for
a while we will begin setting up
migration teams to help move customers
into our own IP space and out of all of
our carriers space.
We will be doing some routine
maintenance to our mail servers today.
We have already completed several core
upgrades to our backup mail server, and
have updated filters, as well as the
spam filtering system on the backup
server. As the backup server has proven
to be quite stable after these upgrades
we will be performing upgrades to
mail.mia.net as well later this
evening. The outage is not expected to
last more than 20 minutes. No mail will
be lost or bounced, however mail
delivery will be slowed and access to
mail will be intermittently available.
We will be moving our current
C&W connections and IP space to New
Edge Networks. We are also exploring
direct NxT or DS3 connections to either
the C&W POP in Chicago, connections
to SBC, or a direct connection to the
Chicago Ameritech NAP which would give
us the ability to buy transit from a
number of carriers that peer there. In
any even please be assured that our
network and connectivity will continue
to both excel and expand amid the
continuing disasters in the Telcom
industry.
SBC Ameritech has completed the
install of Fiber and DS3 equipment in
our facility and server room. We will
have support for 4 DS3 connection once
the fiber is turned up. We expect Turn
up in Mid November.
We have updated all MRTG Graphs for
our routers and backbone connections on
the main
infopages.mia.net web
site. We have finalized the move of our
remaining T1's to our new router, and
our connections to Savvis should now be
balanced properly. We also corrected a
small routing configuration error
yesterday afternoon that was affecting
about 1/3 of overseas traffic.
We have added a new Looking Glass
page for our Cisco 7513 so you can run
pings and traceroutes directly from our
7513 as well now. We have also added
MRTG graphs for our Savvis connections.
As planned our new Savvis T1 was
installed successfully and is up and
running. All customers that had
ethernet port connections to our
network have been moved to our new
Cisco 7513 without issue. Finally our
new 208v 30AMP APC 2200 Rack mount UPS
systems arrived today and have been
installed. We will have them wired in
an live some time next week. That's it
for upgrades for a while until the
digging starts for our Fiber and DS3,
as well as additional power. Have a
nice weekend!
We will be turning up our new Savvis
pipe at 3PM CST today. We will also be
moving our other Savvis pipe to our new
router, as well as moving a few other
PTP T1's on our network that link two
of our POPs to our NOC. Beyond that we
will also be moving customer cross
connects and ethernet connection to the
new router as well. No down time of our
network in general is expected, and all
customers being moved should barely
even notice the move. Once this is done
we are taking a break!
Savvis has given the all clear for
turn up of our new Savvis T1. The
circuit tests out fine from end to end
and is ready to be provisioned. We are
tentatively scheduled to turn up the T1
tomorrow between 10AM and 5PM. There
should be no interruption in services
while this circuit is turned live.
All planned wiring has been
completed by our electricians for our
additional 20 and 30AMP 208v service in
the server room. There is still some
slight work to be done, however this is
the last leg of work on the current
service. We expect to have another
200AMP service installed in our server
room prior to the end of summer. No
outages were present during the work.
SBC Ameritech installed our
additional Savvis T1 today. We awaiting
notification from Sprint and Savvis as
to the turn up date. We expect to have
this circuit operational in the next
week or so. We have been testing out
new Cisco 7513 router and will be
installing the T1 and ethernet modules
midweek and hope to have Savvis and
Ameritech circuits moved to the new
router by the end of the week.
SBC Ameritech and hi-cap fiber techs
will be in our offices all day today
planning our new DS3 installation. All
planning for the upcoming digging has
been completed and we have prepared a
rack for the DS3 equipment which will
be installed in the coming weeks. The
addition of fiber to our NOC will not
only secure greater redunancy and
expansion capabilities for our network,
but will allow customers to order their
own circuits and get cross connects to
the various networks we have at our NOC
in the future. We currently have
multiple redundant connections to
Savvis, Cable and Wireless, PSI/Cogent
and Ameritech.
In our continuing efforts and
commitment to expansion of our network
we are adding a Cisco 7513 in addition
to our Cisco 3640 routers. This will
give us added redundancy and better
control over our network, BGP, and
customer networks. We expect to have
our new Cisco installed in the coming
weeks. There should be no noticeable
down time as we transition to the 7513.
The router has dual power plants which
will be connected to dual and separate
UPS systems for added up time and
redundancy. It also sports redundant
processors as well.
That's right, Friday July 5th, 2002
we will be officially turning up a new
Savvis T1. We expect turn up at 10AM
CST US. While we will be closed for the
Holidays (July 4th through 7th), we
will be here to turn up the T1 Friday.
30 days from now we expect another T1
to Savvis to be turned up. Also, we
have ordered a DS3 from Ameritech which
we expect to be installed in about 18
weeks. We do not expect the additional
T1 turn up to cause any outages and
have taken measures to turn up the
circuit on a different and isolated
router on our network initially. If
nothing else, you should notice an
improvement in network performance.
If you have any questions, or
concerns, please let us know.
Again, we will be closed July 4th -
July 7th in honor of our Nations Birth.
Wherever you are, have a safe and happy
holiday, and since 95% of our customers
are located outside the US, thank you
for your support as a customer, and
your kind words and support offered
after the events of September 11th.
Here's to a safe and happy July 4th.
On call tech are available over the
holiday using the standard methods of
contact.
June 20-02 (3:41 PM
CST)
We are currently suffering from a
partial outage. We have lost all our
connections to Cable and Wireless,
along with several T1's. We are working
with the our providers and the telco to
find the problem. There is no ETA on
this problem. While we are still
connected to the internet via on last
provider, there will be mass latency as
all traffic is being funneled out other
routes. We apologize for any
inconvenience.
May 31-02 (4:41 PM
CST)
So far our PSI connections seem to
be responding at normal capacities.
Again, we will continue to monitor this
situation as we are logging all CRC
errors at our routers to try to find a
pattern to the intermittent problems on
this circuit. In any event all systems
functioning perfectly at the moment.
May 31-02 (10:01 PM
CST)
Ameritech Hi-Cap Techs are in our
NOC testing the PSI circuits. So far
after 30 minutes of testing the line
looks clean. We will continue to test.
There will be a slight bit of latency
while testing takes place.
May 30-02 (3:05 PM
CST)
Our connection to PSI have been
flapping all day long. This means a bit
of latency as BGP sessions go up and
down and as traffic is routed out our
other providers. Ameritech is currently
working on the circuit which appears to
have some sort of problem at the CO.
There is no ETA.
May 29-02 (1:00 PM
CST)
Bella Mia to add connections to
Savvis. This makes Savvis our 4th
provider. Expected install is Late
June.
March 5-02 (7:05 PM
CST)
As of 7:00PM CST our PSI.NET
connection are live again. Total
downtime was 250 minutes. So far all
BGP sessions with PSI are coming back
on line. We expect full routes by
7:10PM. Latency should now subside as
everything is back to normal.
March 5-02 (5:15 PM
CST)
Ameritech has made contact with PSI
as of 5:02PM CST. Ameritech says there
is a cabling issue in the CO (Central
Office) in Milwaukee. There is
currently no ETA either from Ameritech
or PSI. We are showing a current down
time of 150 minutes at this point.
Again, all other connections to the net
are stable and responding. We will keep
you updated as usual throughout the
evening.
March 5-02 (4:41 PM
CST)
As promised our new 1 hours status
notification from PSI.NET returns the
same response as previously noted.
Ameritech has not returned an ETA to
PSI. We have been instructed to call
PSI.NET again at 5:05PM for new
information on the ETA.
March 5-02 (3:41 PM
CST)
PSI.NET tells us that our
connections to their Milwaukee POP went
down at 2:49PM CST. They have been
unable to resolve the issue and believe
it to be a telco problem and have
contacted Ameritech HiCap Repair. There
is still no ETA, but we should have
more news at 4:41PM. Again, our network
is up and operational and BGP is
functioning. However their may be
periods of latency and difficulty
reaching some sites. We apologize for
any inconvenience.
March 5-02 (3:17 PM
CST)
Our connections to PSI.NET are
currently down, however BGP is doing
its job and our connections to Cable
and Wireless as well as Ameritech are
currently functioning normally. There
may be periods of latency as
connections are re-routed
automatically, or until service is
restored. No ETA is available.
February 12-02 (8:17 PM
CST)
Stats.mia.net is back on line.
Apparently BSD could find one of the
NIC cards in the box, but could not
find its IRQ. After doing several tests
we finally pulled the NIC card that was
having trouble and ran a pencil eraser
across the contacts, reinstalled it and
it came back up. Go figure. Never a
dull moment here!
February 12-02 (5:02 PM
CST)
We are still having problems with
stats.mia.net. With this in mind all
customer graphs will be off line as we
seek a solution. Sorry for any
inconvenience.
February 12-02 (11:58
AM CST)
stats.mia.net is currently down. BSD
seems to have lost its ability to find
one of the NIC cards in this box. We
have been working on this problem since
11PM yesterday. NO ETA.
January 25-02 (11:36 AM
CST)
We are upgrading our EIMS server to
version 3.1 today. There will be a
momentary outage of the mail servers
while we perform this upgrade. We
apologize for any inconvenience.
January 15-02 (11:36 AM
CST)
We have replaced the battery packs
on one of our APC Rack mount UPS's. No
down time was experienced as the units
are hot swappable. We have purchased
spares so that we do not run into this
problem again.
January 10-02 (5:29 PM
CST)
More wonderful changes/updates to
our network. We have been busy today:
1. Moved old "alpha.net"/PSI
206.190.23.x IP range out of the main
IP configs on our core router and moved
it to a secondary.
2. Moved the rest of our Portmasters
out of the old 206.190.23.x block
3. Reconfigured OSPF on our
Portmasters and core routers
4. Updated Traffic Graphs for
Portmasters
5. Reconfigured RADIUS to recognize
all changes
6. Replaced Hard Drive on the new
"jer.mia.net" DNS server due to
possible problem/failure with the
previous one.
7. As we continue to test the new
"jer.mia.net", we have installed Debian
Linux and QDNS with no problems.
8. Ordered replacement and spare
battery packs for our APC Rack Mount
units as one unit failed. This is quite
common, however we had no spares. The
unit is "hot swapable" so we will
replace the battery pack once it
arrives.
January 8-02 (3:10 PM
CST)
We are installing a new DNS server
today. Basically we have built new
hardware and are moving jer.mia.net
(208.146.240.10) to a new box. There
should be no noticeable down time
during this move.
January 6-02 (11:50 AM
CST)
All graphing and statistical
information has been restored for all
colocated servers and hosting servers.
January 1-02 (12:01 AM
CST)
Happy New Year!
December 26-01 (11:00
AM CST)
All graphing for colocated servers
is off line temporarily until we can
isolate a bug in our software upgrade
that occurred late last week. Currently
the traffic shaper is functioning, but
all graphing is off until we can find
the cause of several crashes that have
plagued this latest version of the
software. I apologize for any
inconvenience.
December 21-01 (11:00
AM CST)
All graphing has been reestablished.
The traffic shaper has been replaced
and is back up and running at full
steam with no problems reported at this
time. We will be testing to old
hardware to find out what the cause of
the sparatic and intermittent
instability problems were. Again, I
apologize for any inconvenience.
December 20-01 (5:30 PM
CST)
Thanks to the anal behavior of on of
our suppliers there is a delay in
getting a hardware key for our traffic
shaper replacement.. All graphing for
colocated servers will be off line
until morning when we receive the key.
All traffic will keep flowing without
interruption however. Sorry for any
inconvenience.
December 20-01 (11:30
AM CST)
We are currently installing our new
traffic shaper to replace the old one
which has had numerous hardware issues
that have caused intermittent outages
at various time over the last several
months. The new system should be up an
running before days end. In the mean
time all traffic reporting and graphs
will be down.
December 14-01 (11:44
AM CST)
We had a switch and router failure
simultaneously.. Not sure which was the
true culprit at this point. If all else
fails (which it did) we just rebooted
everything. Looks like everything is
back up and running again. We apologize
for any inconvenience this may have
caused.
December 13-01 (11:44
AM CST)
We will be installing our CSU's for
the new T1 Friday December 14th, and
subsequently turning up the new
circuit.
November 15-01 (11:30
AM CST)
Our new T1 is due to be installed on
November 28th, 2001.
October 30-01 (4:00 PM
CST)
We have ordered another T1 to link
to a new POP. The T1 will create a
"round-robin" loop between our other
POP's and main NOC. This is being done
both to deliver extra bandwidth to the
new POP as well as to ad another line
of redundancy system wide. There will
be no noticeable down time associated
with this new circuit.
September 24-01 (1:00
PM CST)
UPDATE:
Network Outage
After 24 hours of lost connectivity
to one of our backbone providers we
were just informed by Ameritech that
they have found the problem. Apparently
there is a bad set of transmit pairs in
a cross box between our Facility and
the Lake Geneva Central office. CSG or
the Cable Splicing Group has been sent
to fix the problem and replace the
pairs. There is still no ETA.
September 23-01 (1:00
PM CST)
Subject:
Network Outage
There is a partial network outage
that we are tracking starting at around
1:00PM CST. It would appear that one of
our connections to the internet is not
working. Since we have other providers
and other connections to the internet
our network continues to run. There
will however be slight latency as the
other pipes carry to load. Also there
is a small block of IP's which are
currently unroutable out our other
providers. This is causing some
problems for some customer servers in
this block, and several hosted web
sites on our network. We have been
migrating all customers in this block
into a new block for several months
now. We are stepping up the time table
for this migration and will be doing so
immediately. Some sites may be
unreachable to the outside world for
several hours while this new
information propagates around the
world. Thank you for your
understanding.
September 18-01 (10:00
AM CST)
Subject:
UNKNOWN WIDE SPREAD ATTACK ON THE
INTERNET TODAY
Attention colocation customers. If
you are using an Microsoft Branded
Windows based server please make sure
you read this completely.
If your box is affected, we strongly
urge you to patch it once a new patch
becomes available.
http://www.cert.org/current/current_activity.html
Increase in Port 80 (HTTP) scanning
activity
This morning (September 18th) the
CERT/CC started receiving reports of a
massive increase in scanning directed
at port 80 (HTTP). Reports indicate
that this scanning activity is
attempting to exploit systems
previously compromised by Code Red II
and/or the sadmind/IIS worm as well as
other known vulnerabilities in
Microsoft Internet Information Server
(IIS). Please see CERT Vulnerability
Note VU#111677 for information on the
type of vulnerability being exploited.
The following is a log excerpt of
this scanning activity:
GET /scripts/root.exe?/c+dir
GET /MSADC/root.exe?/c+dir
GET /c/winnt/system32/cmd.exe?/c+dir
GET /d/winnt/system32/cmd.exe?/c+dir
GET
/scripts/..%5c../winnt/system32/cmd.exe?/c+dir
GET
/_vti_bin/..%5c../..%5c../..%5c../winnt/system32/cmd.exe?/c+dir
GET
/_mem_bin/..%5c../..%5c../..%5c../winnt/system32/cmd.exe?/c+dir
GET
/msadc/..%5c../..%5c../..%5c/..\xc1\x1c../..\xc1\x1c../..\xc1\x1c../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..\xc1\x1c../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..\xc0/../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..\xc0\xaf../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..\xc1\x9c../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..%35c../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..%35c../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..%5c../winnt/system32/cmd.exe?/c+dir
GET
/scripts/..%2f../winnt/system32/cmd.exe?/c+dir
The CERT/CC has also received
reports of a possibly new piece of
malicious code named "readme.exe" being
sent via email. Preliminary analysis
indicates that this file may be related
to the increase in port 80 scanning
activity.
Sites are encouraged to verify the
state of security patches on all IIS
servers and email client software.
Administrators may also want to add
filters to mail servers to block the
"readme.exe" attachment. In addition,
sites may wish to notify users of the
existence of "readme.exe" and its
potential threat.
September 12-01 (7:00
AM CST)
Subject:
Update on the situation and status of
networks in the US
I just wanted to update everyone of
our colo customers and web hosting
customers on our Network's and
subsequent "Network" status. We are
currently running at 100%, and have
been since this terrible atrocity
committed upon our great nation. We are
situated in the Midwest, far away from
major cities like Chicago and
Milwaukee. The networks that we connect
to are also 99.9% intact.
There are however many problems as
you might guess on the East Coast
relevant to damage beyond belief.
However we are as of yet relatively
unaffected. I just wanted to assure all
of you that we are up and running here,
as are most networks and businesses. We
are at work, we are picking up the
pieces and we are moving on. I know
that this type of assurance is quite
important to several of our customers
since 95% of our customer base is
located "outside" of the US. We are a
tough nation that has had to fight for
a long time to gain, and maintain
freedom, and will continue to do so.
For those of you outside of the US, the
mood, while somber is quite positive
and focused on doing all we can to help
everyone affected.
In any event, everything is running
quite smoothly at this hour and has
been since this tragedy began. If you
have any questions, concerns, or
even need to try to contact
relatives in the US, we are here and
would be glad to relay messages either
via the Internet or phone here in the
US.
Also, if you do have trouble getting
to corresponding networks that crisis
cross their way from other parts of the
world to your servers or web sites here
at Bella Mia, Inc., rest assured that
they are up and running. We have set up
extra monitoring to insure that we
catch any outage, that you may not be
able to.
The best thing we all can do right
now is take some time to pray for those
who were/are directly affected by this
diabolical attack.
God bless everyone involved, and God
Bless America!
Regards,
Jeremy Anthony Kinsey
VP Network Operations - Bella Mia, Inc.
September 11-01 (8:58
AM CST)
Subject: Terrorist
Attacks
We at Bella Mia, as well as other
ISP's across the nation are urging dial
up customers to get off the phone
lines. There was a major terrorist
attack in New York and Washington today
that took the lives a thousands of
innocent people when 3 separate planes
flew into the World Trade Center in
NYC, and the Pentagon in Washington.
The damage and death toll is
devastating.
Several people may have family and
friend in New York so again, we are
urging all customers that use dialup to
connect to the internet to please stay
off the lines. Your efforts will help
keep the phone lines open for calls in
and out of New York. We have
experienced trouble calling locally due
to the large call volumes from this
travesty on our great nation.
If you are using Wireless, or
another dedicated connection, there is
no need to stay off the internet. If
you must use dialup, be brief and use
email to communicate with your loved
ones.
Thank you for your help during this
sad day.
Please keep all these people in our
prayers today.
Due to the tragedies in New York and
Washington DC, customers may experience
problems in these areas. Also, all news
sites, such as www.cnn.com, are
experiencing high volumes which may
prevent user access.
All issues are being tracked in at
our NOC.
Again, KEEP THE PHONE LINES CLEAR!!!
Thank you,
The Bella Mia Family
August 28-01 (3:38 PM
CST)
Our 2 New T1's
finally went up today without a hitch,
adding 3Mbits of Bandwidth to our
network. There was no noticeable down
time or latency during the upgrade. All
BGP sessions came back up within a few
minutes and our peering session is now
active with Cable and Wireless. The
additional bandwidth will allow for
future growth as well as allow our
current customers the speed and
performance they have come to
appreciate. We will be doing some
testing later this evening by pulling
down circuits one at a time and turning
them back up to insure that our load
balancing and fall over works as
intended.
If you have
any questions or concerns, please let
us know.
August 28-01 (12:58 PM
CST)
***URGENT***
Subject: Network Outages and Latency
Today
We will be installing 2 T1's today
adding 3Mbits of Bandwidth to our
network. While this will not be a
complete outage, there will be a slight
amount of latency as our routers fall
over to our other upscale providers
while we move/add our new circuits.
Also, during reconfiguration of our
BGP and load balancing our Core Router
may be rebooted causing momentary down
time.
If all goes well the upgrade should
be no more than 1 hour. Downtime should
be limited to a matter of minutes only.
This upgrade is a long time in coming
and should provide increased speed and
better routes in and out of our
network.
What does all this mean to you?
During the upgrade things will slow
down a bit, and there may be some spot
outages. Delays in mail, or access
outside of our network. Again, the
downtime will be quite limited.
If you have any questions or
concerns, please let us know.
August 24-01 (11:58 PM
CST)
As of 8:00PM CST we have fixed our
DNS servers and all DNS traffic and
queries should not be active again to
our network. It looks like the DNS DOS
was caused by the "orbz" list people's
DNS servers.
Hers is the jist of the problem:
- Our DNS server starts trying to
resolve something in the
inputs.orbz.org zone.
- It creates an internal list of
servers to query, and starts its way
through the list.
- Later, it modifies this list
without checking the data integrity.
- It continues to try to use the
list as if the list had not been
modified.
- Crash.
We're not fully able to quantify
what's wrong with the inputs.orbz.org
servers - Sigfus (the engineer who
worked on our system on Friday) says,
"the setup over there is weird." All
they can tell you us, the answer we get
from a.ns.inputs.orbz.org contains a
refusal to answer; b.ns.inputs.orbz.org
doesn't answer at all at the moment.
August 24-01 (4:58 AM
CST)
We are experiencing some kind of DOS
(Denial of Service) attack on our DNS
servers. This problem is causing our
DNS servers to crash. The only way we
can mainatin stability at the moment is
to block all incoming UDP packets from
the outside world to our network. This
will cause DNS services for any of the
domains that we (Mia.Net) hosts to be
unreachable to the outside world. We
are currently working with our software
vendor to find a resolution.
July 11-01 (4:32 PM
CST)
We have finally received the license
keys for upgrades to our traffic shaper
and graphing machines which should be
installed Thursday. The upgrade should
eliminate the crashing problem on this
box. Once this fix is applied the
graphs should not die in the future.
Again, sorry for any inconvenience this
has caused. Also, we have moved all of
our switches to an internal/fake IP
addressing scheme. This should insure
that no one outside our network can
attempt access to any of our switches.
July 1-01 (4:32 PM
CST)
All customer graphs are finally
running again, and stats.mia.net is
fixed. We have also set up spanning
tree support on all of our switches. If
for some reason stats.mia.net or our
traffic shapers go down, network
connectivity will be maintained. We
have tested the fall-over capabilities
of this system, and found the change to
be instantaneous.
We also upgraded our core router to
IOS 12.1 only to find that Cisco's new
IOS killed one of our serial ports
taking down a PTP T1 link. We reverted
back to the old flash and have notified
Cisco's routing group. It's been a
hellish, crazy upgrade day.
June 30-01 (8:16 AM
CST)
All customer graphs will be down
indefinitely until we can find out what
is causing problems with stats.mia.net.
Again, graphing is disabled for all
colocation servers due to an upgrade on
stats.mia.net and stats2.mia.net,
however stats2.mia.net is unaffected by
this. We apologize for any
inconvenience.
June 27-01 (10:24 AM
CST)
All customer graphs should be
restored. Again, graphing was off for
all colocation servers due to an
upgrade on stats.mia.net and
stats2.mia.net. We apologize for any
inconvenience.
June 26-01 (9:10 AM
CST)
We are recompiling the BSD kernels
on stats.mia.net and stats2.mia.net.
Both of these boxes run our traffic
management, backup routing, and a few
other tasks. stats.mia.net is currently
down, so you may not be able to see
your traffic graphs, or they may look a
bit strange.
June 21-01 (10:36 AM
CST)
According to one our techs... From
what we can tell,
the-isp-formerly-known-as alpha.net
dropped it's link to above.net
yesterday (June 20th, 2001). We do not
think alpha has any other connections
left, so now everything is going out
their PSI pipe.
As a result our network traffic was
pretty out of balance because of this.
We are prepending Cable and Wireless's
AS number onto all routes we get from
them. That should balance things
better. Once we get the our additional
C&W T1s, we will drop all the stuff
for prepending and weighing things
differently and the our core cisco can
do things with an un-modified routing
table.
We apologize for any inconvenience
or slow downs this may have caused.
June 15-01 (2:26 PM
CST)
We powered down home.mia.net today
to install more RAM. There would have
been a slight outage of not more than 2
minutes for customers hosted on this
box. We apologize for any
inconvenience.
June 13-01 (6:56 PM
CST)
We still do not have a definitive
install date for our new Cable and
Wireless T1's. They are about two weeks
over due. We are hopeful they will be
installed by the end of next week.
June 12-01 (12:01 PM
CST)
We powered down our core router to
install new WIC cards for our new Cable
and Wireless T1's due in any day now.
There would have been a 1-2 minute loss
of connectivity during this upgrade.
June 7-01 (5:01 PM
CST)
Well back hoe harry and our moronic
neighbors in the industrial park
decided today would be a good day to
dig through a cable run in front of our
building. While this (luckily did not
affect any of our T1's since they take
a different path) it did affect our
voice and fax lines. Our phone lines in
the office are still pretty dead as we
speak. HiCap is on site doing an
emergency splice. They are however only
splicing services for Sprint and
Telecorp who have back haul T1's going
through these lines. Things are a mess
right now, but again, it has not
affected our "data" network. All
servers are live and going strong.
Someone is going to pay through the
roof for this!!! We have instructed our
attorneys to stop construction/digging
until further notice. We have also
contacted the City Building Inspector
and the City Engineer who signed off on
this stupid project.
June 7-01 (8:21 AM
CST)
Ameritech installed 2 T1's today for
our new Cable and Wireless backbones.
We expect them live next week some
time.
June
1-01
For some strange reason www.mia.net
died today. Webstar just quit. We spent
several hours trying to diagnose the
problem and could not find a reason for
the crash. This is quite odd since this
server never crashes. As of 12:01PM
www.mia.net is back up, and running
normally. We will keep a close eye on
this server to insure that stability is
maintained. We apologize for any
inconvenience.
May
23-01
I am happy to announce that our
Server Room Expansion is near
completion. The walls have been
removed, new paint, and a new floor
have been installed. Additional power
has also been installed. We are
currently installing racks to add
additional capacity for approximately
200 servers. We still do not have an
install date on our Liebert system. We
are hopeful it will be installed in
June. Also, in the first weeks of June
we will be installing our 2 new Cable
and Wireless T1's. Stay tuned.
May
22-01
All of our Mac based Web Hosting
servers have been upgraded to Rumpus
1.3.4. Version 1.3.4 corrects a couple
of problems, the most important of
which is a possible system hang that
can be caused by specific illegal input
from an FTP client. Because the crash
could potentially be used in a denial
of service attack, Maxum will not
discuss the specific method required to
reproduce the problem. However, both
Maxum and Bella Mia, Inc. urge all
Rumpus 1.3 users to perform this free
upgrade. If you have a Mac Server
colocated here please download the
update from
www.maxum.com/rumpus
May
10-01
One of our BSD boxes that controls
the traffic graphs had a problem. The
trouble has since been corrected.
Customers would have noticed a lack of
traffic on their graphs during a 24
hour period while we fixed the problem.
No network connectivity was lost. This
was merely a cosmetic issue.
May 2-01
Looking Glass on our Cisco 2501
should be functional again:
http://stats2.mia.net/lg.html
May 1-01
Our New T1's from Cable and Wireless
have been pushed up to June 1, 2001 for
tentative install.
April
26-01
Well it looks like one of our
upscale providers that carries
transport to the NAP in Chicago went
bankrupt the other day. They did not in
fact have a fiber cut, or switch
problem as they had indicated
previously. Their provider shut them
off. Luckily we had another set of PTP
T1's at our remote POP where this
outage occurred. These T1's have been
running for several months, but have
yet to pass traffic. It took a bit of
reworking, but we moved everything at
this remote POP on to our network, and
all is functioning at 100% again.
NS2.MIA.NET is back up, however on a
new IP now, which has been changed in
the DNS, and with our Host Registrar.
The same can be said of one of our
backup mail servers which also resides
at this remote POP. In any event this
outage did not affect any traffic to
customer servers, or hosted web sites.
Anyway, NS2 and MX Mail services at
this POP are back up.
April 25-01 (2:55PM
CST)
We now have a tentative install date
of July 1, 2001 for two NxT T1's
(3Mbits) from Cable and Wireless.
April 25-01 (2:25PM
CST)
OUTAGE:
Some moron with a third
grade education and a back hoe decided
it would be a good idea to forgo
calling Diggers Hot line and cut four
OC48's that feed one of our POPs. What
does this mean for colo/web hosting
customers and our network? Not much.
Only ns2.mia.net is down at this time
along with a backup PTP feed. Pays to
run BGP and have multiple connections
to the net, as well as multiple DNS
servers. Anyway, there is no ETA on the
damage caused by back hoe Harry. I do
understand that this problem affects
Illinois, Indiana, and Wisconsin
however.
April
11-01
Work has been stepped up on our
server room expansion project. We
should have the room complete in about
2-3 weeks. New power will be added next
week, including power for the new
Liebert system. We have installed a
Window on one of the outside walls of
the server room so that visitors can
peer in. It should be pretty cool once
the room is complete, and filled of
course. Our new Cable and Wireless T1
is also on order.
April
3-01
Our new server room expansion
project is now underway. We will be
adding about 500 sq ft of space to the
current room and are having Liebert
technicians assess the space for a new
cooling plant and environmental control
system. We will have space for an
additional 200 CPU's by Summer 2001.
April 1-01 (1:07 A.M.
CST)
All MRTG graphs for customer servers
and or hosted web sites are now back on
line. No polling data was lost during
the outage. Thank you for your
patience.
March 31-01 (3:28 P.M.
CST)
All MRTG graphs for customer servers
and or hosted web sites are currently
off line. We are repairing the system
that maintains the graphs as a result
of this mornings router failure. We
apologize for any inconvenience and
hope to have graphing restored shortly.
March 31-01 (12:18 A.M.
CST)
OUTAGE:
At approximately
12:18AM(CST) one of our border routers
failed. This particular router controls
a great deal of customer routes,
including some internal and external
parts of our network. While our entire
network did not go down, we did loose a
great deal of routes for various parts
of our network, including all customer
aggregation points, ie., colocation
servers. After manually bypassing the
router and restoring network stability
we have examined the problem router and
can find nothing wrong. We have
rebooted the router and checked all
configs and have since brought the
device back on line. Some customers may
have noticed that reachability to our
network as a whole was patchy, and in
some places completely unreachable. We
will monitor this router closely
throughout the weekend to insure that
this does not occur again. We apologize
for any inconvenience this may have
caused.
March
29-01
Thanks to the help from Chris Buxton
from Men and Mice we have successfully
changed all "legacy" domains/zones that
contained homer.alpha.net and
helen.alpha.net to the new PSI.NET
secondary name servers. We have also
reconfigured ns2.mia.net to reflect
these changes and have subsequently
rebooted both ns.mia.net and
ns1.mia.net to reflect these updates as
well.
March
27-01
We reset our BGP sessions yesterday
afternoon in an attempt to better
balance our traffic. This would have
caused a slight delay in connectivity
lasting not more than a minute. We will
keep you updated as we continue to add
and manage more connectivity.
March
21-01
STATUS
UPDATE: Our router upgrade
was successful. Total down time was
limited to about 5 minutes, (the time
it took for our BGP sessions to come
back up.) We will be configuring our
new T1 later today. No noticeable down
time should result from these
configurations.
March
20-01
PLANNED
OUTAGE: Welcome to Spring!!!
We will be powering down our core
router at 12:00AM CST to install a new
T1. This will be a system wide outage
of our network and should not last
longer than 5 minutes. We apologize for
any inconvenience.
March 14-01(2:31PM
CST)
core-bmia-burl-wisp-s0-0-1cw.mia.net
was sucessfully installed today. Turn
up date for the T1 on this router is
March 22nd.
March 13-01(2:52PM
CST)
We are adding another T1 to create a
PTP link to a POP of ours. The link
will help facilitate redundancy at this
POP where we have various backup
servers, such as DNS, Mail, etc.,
located at. The install date is set for
March 22nd. At that time an additional
Core router
(core-bmia-burl-wisp-s0-0-1cw.mia.net)
will be turned up. This install will
not affect connectivity, nor will there
be any down time during installation
and configuration.
February 21-01(11:20AM
CST)
We have been doing some security
checks do to a recent hacker intrusion
on one of our customer colocated
servers. We recently checked the ftpds
running on a couple ips near the
customer server that was hacked. All
colocation customers might want to
check to find out if your ftp server up
to date with the recent security
patches and bug fixes, and if no, fix
it and check for any intrusions.
February 6-01(11:20AM
CST)
A system wide upgrade of the
following Bella Mia Domain Name Servers
(DNS) was performed today:
jer.mia.net
ns.mia.net
ns1.mia.net
ns2.mia.net
January 29-01(11:50AM
CST)
MRTG Graphs for customer bandwidth
utilization are back up.
January 29-01(10:50AM
CST)
MRTG Graphs for customer bandwidth
utilization are down currently. We are
doing a bit of maintenance due to lasts
nights unexpected problem with our
customer aggregation router. All graphs
should be back to normal by days end.
January 28-01(6:50PM
CST)
One of our core routers took a dump
for some reason, which is yet
undetermined. This would have caused a
loss of reachability for some customer
servers, and routing on our network. We
have since rebooted the the troubled
router and stability has been restored.
January 24-01(1:50PM
CST)
ns.mia.net (208.146.240.194) is back
up and fully operational.
January 24-01(11:00AM
CST)
ns.mia.net (208.146.240.194) will be
off line for a hard drive replacement
and system upgrade.
January 21-01(3:00PM
CST)
Our new core router to Nap.Net/BBN
Planet is now operational.
(core-bmia-burl-wisp-dc0-1cw.mia.net)
January
1-01
Welcome to the New Century! January
2001. Last months news has been
archived to
the
following link.
