NOTICE DATE:
April 9, 2008
NOTICE TYPE: M-A031208-06 UPDATE: Outage/Retail
Processing
CLASSIFICATION: Public
SHORT DESCRIPTION:
Additional impacts due to the retail transaction processing outage that
occurred on March 12, 2008.
INTENDED AUDIENCE:
CRs and TDSPs
DAY AFFECTED:
March 12, 2008 through March 21, 2008
LONG DESCRIPTION:
ERCOT has been made aware of additional impacts due to the
retail
transaction processing outage that occurred on March 12, 2008. Some 867_03
files were not processed. The files were received by ERCOT and
acknowledged but were not included in the files created for data loading into
the Settlements and Billing application. Both IDR and NIDR files were among
those affected.
ERCOT is working to identify the unprocessed
files so that they may be reprocessed. As a result of reprocessing,
Market Participants may receive duplicate 867_03 forwards. Following are
initial counts of impacted 867_03:
|
|
Count |
Files |
|
NIDR |
12504 |
1 |
|
IDR |
117 |
15 |
The outages and Retail processing
issues encountered were the result of the following:
1)
Failed production release of SCR745
(Retail Market Outage Evaluation and Resolution) - Due to slow performance and
crashes, an outage outside of the standard maintenance window was utilized to
rollback to pre-implementation state.
2)
Storage hardware failures – A
project to replace aged hardware has been in progress since late 2007 and is
scheduled for implementation in May.
3)
Human error – Operating
procedure revisions have been made to prevent future occurrences as well as
education and training.
ADDITIONAL INFORMATION: ERCOT
Retail Account Manager’s will provide all affected Market
Participant’s with a list of impacted 867_03s. The lists are
expected to be ready by Thursday, April 10, 2008 after 9:00 AM.
CONTACT: If you have
any questions, please contact your ERCOT Account Manager. You may also call the
general ERCOT Client Services phone number at (512) 248-3900 or contact ERCOT Client
Services via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution
list that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/DG
From: ERCOT Client Relations
Sent: Thursday, March 20, 2008
5:42 PM
To:
[log in to unmask];
[log in to unmask];
[log in to unmask]
Cc: [log in to unmask];
[log in to unmask]; [log in to unmask]; 1 ERCOT Client Service Reps
Subject: M-A031208-05 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 20, 2008
NOTICE TYPE: M-A031208-05
UPDATE: Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION:
Additional transactions have been impacted by the retail transaction
processing outage that occurred on March 12, 2008.
INTENDED AUDIENCE: CRs and
TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: The
following transactions were impacted as a result of the retail transaction
processing outage on March 12, 2008.
Details
on impacted transactions:
·
867s - ERCOT had errors during the
reprocessing efforts on March 18, 2008 and March 19, 2008. ERCOT will be
checking with the two (2) TDSPs again for approval to reprocess.
Updated counts
|
|
DUNS |
Count |
|
867s |
10994067400 |
10,008 |
|
867s |
957877905 |
106,477 |
·
ERCOT analysis has identified an
additional set of 867 forwarding delays.
Updated counts
|
|
DUNS |
Count |
|
867 forwards |
1039940674000 |
19628 |
·
ERCOT Retail Account
Manager’s will provide all affected Market Participant’s with a
list of impacted 867 transactions (both the reprocessing and the forwarding delays)
after 8:00 AM on Friday, March 21, 2008.
·
814s - ERCOT has completed the
ERCOT Initiated MarkeTrak issues for 814s that have been identified as not
processing to request approval to reprocess. The final count of impacted
transactions was 139.
·
ERCOT Retail Account
Manager’s have been provided a list of MarkeTrak issue numbers and will
provide those to the affected Market Participants.
·
997s - ERCOT is reprocessing
current missing 997 requests via MarkeTrak issues and performing additional
analysis for any potential impacts.
ADDITIONAL INFORATION: If Market
Participants are missing transactions, ERCOT requests that a MarkeTrak issue be
filed, with an attached spreadsheet, for ERCOT to research.
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT
Client Services phone number at (512) 248-3900 or contact ERCOT Client Services
via e-mail at [log in to unmask].
If you are
receiving e-mail from an ERCOT distribution list that you no longer wish to
receive, please follow this link in order to unsubscribe from this list: http://lists.ercot.com.
JA/DG
From: ERCOT Client Relations
Sent: Tuesday, March 18, 2008 5:21
PM
To:
Cc:
Subject: M-A031208-04 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 18, 2008
NOTICE TYPE: M-A031208-04
UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT has
determined that a subset of transactions on March 12, 2008 was not processed
through ERCOT systems.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT has determined
that the retail transaction processing issues, which occurred on March 12,
2008, were the result of human error. A slow-down on the application
server was being investigated and was believed to be a hardware issue.
During the investigation, the batch process was mistakenly restarted while one
was already running. This caused errors with the transaction processing
and corrupted log files.
Reason for delay in communication to the Market: The
validity of the exception queries that run against the logs was in
question. The effort continues to be a challenge due to the volume of
data and the anomaly of the problem that occurred with log files being
corrupted or missing as a result.
ADDITIONAL INFORMATION: Details on
impacted transactions:
867s
- ERCOT Initiated MarkeTrak issues were submitted to two (2) TDSPs and ERCOT received the approval to
reprocess (on March 18, 2008). ERCOT has begun to reprocess and forward
the 867s.
|
|
DUNS |
Count |
|
867s |
1039940674000 |
10,008 |
|
867s |
957877905 |
106,477 |
ERCOT
will continue analysis to identify any additional 867 forwarding delays.
814s
- ERCOT is logging ERCOT Initiated MarkeTrak issues for 814s that have been
identified as not processing to request approval to reprocess. The estimated
count is ~ 150 transactions.
997s
- ERCOT is reprocessing current missing 997s requests via MarkeTrak issues and
performing additional analysis for any potential impacts.
ERCOT
Retail Account Manager’s will provide all affected Market Participant’s
with a list of impacted 867 transactions after 2:00 PM on Wednesday, March 19,
2008. 814s will
be communicated as ERCOT initiated MarkeTrak issues.
CONTACT: If you have any questions, please
contact your ERCOT Account Manager. You may also call the general ERCOT Client
Services phone number at (512) 248-3900 or contact ERCOT Client Services via
e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/TR
From: ERCOT Client Relations
Sent: Monday, March 17, 2008 3:58
PM
To:
Cc:
Subject: M-A031208-03 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 17, 2008
NOTICE TYPE: M-A031208-03
UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT experienced
retail transaction processing issues on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT continues to
analyze these processing issues to determine the volume and transactions
impacted. Additional
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT Client
Services phone number at (512) 248-3900 or contact ERCOT Client Services via
e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/JA
From: ERCOT Client Relations
Sent: Friday, March 14, 2008 4:39
PM
To:
Cc:
Subject: M-A031208-02 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 14, 2008
NOTICE TYPE: M-A031208-02
UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT experienced
retail transaction processing issues on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT has
discovered an issue processing a subset of retail transactions as a result of
the outage on Wednesday, March 12, 2008. ERCOT is currently analyzing
these issues to determine the volume and transactions impacted.
Additional
CONTACT: If you have any questions, please
contact your ERCOT Account Manager. You may also call the general ERCOT Client
Services phone number at (512) 248-3900 or contact ERCOT Client Services via
e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/JA
From: ERCOT Client Relations
Sent: Wednesday, March 12, 2008
10:57 AM
To:
Cc:
Subject: M-A031208-01 Outage
– Retail/Wholesale
NOTICE DATE: March
12, 2008
NOTICE TYPE: M-A031208-01
Outage – Retail/Wholesale
CLASSIFICATION: Public
SHORT DESCRIPTION:
This morning, ERCOT experienced issues with the Customer Registration
Application servers.
INTENDED AUDIENCE: CRs
and TDSPs
DAY AFFECTED:
Wednesday, March 12, 2008 from 9:08 AM to 10:15 AM
LONG DESCRIPTION:
This morning, Wednesday, March 12, 2008 from 9:08 AM to 10:15 AM
ERCOT
experienced an issue with the Customer Registration Application servers.
Impacts included Retail Transaction Processing, Texas Market Link Query ESIID
Transaction, eServices Disputes and Service Requests, and MarkeTrak.
ERCOT’s initial assessment
has determined that the root cause is due to a hardware failure.
Additional details will be provided in the ERCOT IT Incident Market Notice
Summary.
CONTACT: If you have
any questions, please contact your ERCOT Account Manager. You may also call the
general ERCOT Client Services phone number at (512) 248-3900 or contact ERCOT
Client Services via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution
list that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
TR/JA