NOTICE DATE: April 9, 2008
NOTICE TYPE: M-A031208-06 UPDATE: Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: Additional impacts due to the retail transaction
processing outage that occurred on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008 through March 21, 2008
LONG DESCRIPTION: ERCOT has been made aware of additional impacts due
to the retail transaction processing outage that occurred on March 12,
2008. Some 867_03 files were not processed. The files were received by
ERCOT and acknowledged but were not included in the files created for
data loading into the Settlements and Billing application. Both IDR and
NIDR files were among those affected.
ERCOT is working to identify the unprocessed files so that they may be
reprocessed. As a result of reprocessing, Market Participants may
receive duplicate 867_03 forwards. Following are initial counts of
impacted 867_03:
Count
Files
NIDR
12504
1
IDR
117
15
The outages and Retail processing issues encountered were the result of
the following:
1) Failed production release of SCR745 (Retail Market Outage
Evaluation and Resolution) - Due to slow performance and crashes, an
outage outside of the standard maintenance window was utilized to
rollback to pre-implementation state.
2) Storage hardware failures - A project to replace aged hardware
has been in progress since late 2007 and is scheduled for implementation
in May.
3) Human error - Operating procedure revisions have been made to
prevent future occurrences as well as education and training.
ADDITIONAL INFORMATION: ERCOT Retail Account Manager's will provide
all affected Market Participant's with a list of impacted 867_03s. The
lists are expected to be ready by Thursday, April 10, 2008 after 9:00
AM.
CONTACT: If you have any questions, please contact your ERCOT Account
Manager. You may also call the general ERCOT Client Services phone
number at (512) 248-3900 or contact ERCOT Client Services via e-mail at
[log in to unmask]
If you are receiving e-mail from an ERCOT distribution list that you no
longer wish to receive, please follow this link in order to unsubscribe
from this list: http://lists.ercot.com <http://lists.ercot.com/> .
DP/DG
________________________________
From: ERCOT Client Relations
Sent: Thursday, March 20, 2008 5:42 PM
To: [log in to unmask];
[log in to unmask];
[log in to unmask]
Cc: [log in to unmask]; [log in to unmask];
[log in to unmask]; 1 ERCOT Client Service Reps
Subject: M-A031208-05 UPDATE: Outage/Retail Processing
NOTICE DATE: March 20, 2008
NOTICE TYPE: M-A031208-05 UPDATE: Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: Additional transactions have been impacted by the
retail transaction processing outage that occurred on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: The following transactions were impacted as a result
of the retail transaction processing outage on March 12, 2008.
Details on impacted transactions:
* 867s - ERCOT had errors during the reprocessing efforts on
March 18, 2008 and March 19, 2008. ERCOT will be checking with the two
(2) TDSPs again for approval to reprocess.
Updated counts
DUNS
Count
867s
10994067400
10,008
867s
957877905
106,477
* ERCOT analysis has identified an additional set of 867
forwarding delays.
Updated counts
DUNS
Count
867 forwards
1039940674000
19628
* ERCOT Retail Account Manager's will provide all affected Market
Participant's with a list of impacted 867 transactions (both the
reprocessing and the forwarding delays) after 8:00 AM on Friday, March
21, 2008.
* 814s - ERCOT has completed the ERCOT Initiated MarkeTrak issues
for 814s that have been identified as not processing to request approval
to reprocess. The final count of impacted transactions was 139.
* ERCOT Retail Account Manager's have been provided a list of
MarkeTrak issue numbers and will provide those to the affected Market
Participants.
* 997s - ERCOT is reprocessing current missing 997 requests via
MarkeTrak issues and performing additional analysis for any potential
impacts.
ADDITIONAL INFORATION: If Market Participants are missing transactions,
ERCOT requests that a MarkeTrak issue be filed, with an attached
spreadsheet, for ERCOT to research.
CONTACT: If you have any questions, please contact your ERCOT Account
Manager. You may also call the general ERCOT Client Services phone
number at (512) 248-3900 or contact ERCOT Client Services via e-mail at
[log in to unmask]
If you are receiving e-mail from an ERCOT distribution list that you no
longer wish to receive, please follow this link in order to unsubscribe
from this list: http://lists.ercot.com <http://lists.ercot.com/> .
JA/DG
________________________________
From: ERCOT Client Relations
Sent: Tuesday, March 18, 2008 5:21 PM
To: [log in to unmask];
[log in to unmask];
[log in to unmask]
Cc: [log in to unmask]; [log in to unmask];
[log in to unmask]; 1 ERCOT Client Service Reps
Subject: M-A031208-04 UPDATE: Outage/Retail Processing
NOTICE DATE: March 18, 2008
NOTICE TYPE: M-A031208-04 UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT has determined that a subset of transactions
on March 12, 2008 was not processed through ERCOT systems.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT has determined that the retail transaction
processing issues, which occurred on March 12, 2008, were the result of
human error. A slow-down on the application server was being
investigated and was believed to be a hardware issue. During the
investigation, the batch process was mistakenly restarted while one was
already running. This caused errors with the transaction processing and
corrupted log files.
Reason for delay in communication to the Market: The validity of the
exception queries that run against the logs was in question. The effort
continues to be a challenge due to the volume of data and the anomaly of
the problem that occurred with log files being corrupted or missing as a
result.
ADDITIONAL INFORMATION: Details on impacted transactions:
867s - ERCOT Initiated MarkeTrak issues were submitted to two (2) TDSPs
and ERCOT received the approval to reprocess (on March 18, 2008). ERCOT
has begun to reprocess and forward the 867s.
DUNS
Count
867s
1039940674000
10,008
867s
957877905
106,477
ERCOT will continue analysis to identify any additional 867 forwarding
delays.
814s - ERCOT is logging ERCOT Initiated MarkeTrak issues for 814s that
have been identified as not processing to request approval to reprocess.
The estimated count is ~ 150 transactions.
997s - ERCOT is reprocessing current missing 997s requests via MarkeTrak
issues and performing additional analysis for any potential impacts.
ERCOT Retail Account Manager's will provide all affected Market
Participant's with a list of impacted 867 transactions after 2:00 PM on
Wednesday, March 19, 2008. 814s will be communicated as ERCOT
initiated MarkeTrak issues.
CONTACT: If you have any questions, please contact your ERCOT Account
Manager. You may also call the general ERCOT Client Services phone
number at (512) 248-3900 or contact ERCOT Client Services via e-mail at
[log in to unmask]
If you are receiving e-mail from an ERCOT distribution list that you no
longer wish to receive, please follow this link in order to unsubscribe
from this list: http://lists.ercot.com <http://lists.ercot.com/> .
DP/TR
________________________________
From: ERCOT Client Relations
Sent: Monday, March 17, 2008 3:58 PM
To: [log in to unmask];
[log in to unmask];
[log in to unmask]
Cc: [log in to unmask]; [log in to unmask];
[log in to unmask]; 1 ERCOT Client Service Reps
Subject: M-A031208-03 UPDATE: Outage/Retail Processing
NOTICE DATE: March 17, 2008
NOTICE TYPE: M-A031208-03 UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT experienced retail transaction processing
issues on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT continues to analyze these processing issues to
determine the volume and transactions impacted. Additional information
will be provided as it becomes available.
CONTACT: If you have any questions, please contact your ERCOT Account
Manager. You may also call the general ERCOT Client Services phone
number at (512) 248-3900 or contact ERCOT Client Services via e-mail at
[log in to unmask]
If you are receiving e-mail from an ERCOT distribution list that you no
longer wish to receive, please follow this link in order to unsubscribe
from this list: http://lists.ercot.com <http://lists.ercot.com/> .
DP/JA
________________________________
From: ERCOT Client Relations
Sent: Friday, March 14, 2008 4:39 PM
To: [log in to unmask];
[log in to unmask];
[log in to unmask]
Cc: [log in to unmask]; [log in to unmask];
[log in to unmask]; 1 ERCOT Client Service Reps
Subject: M-A031208-02 UPDATE: Outage/Retail Processing
NOTICE DATE: March 14, 2008
NOTICE TYPE: M-A031208-02 UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT experienced retail transaction processing
issues on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT has discovered an issue processing a subset of
retail transactions as a result of the outage on Wednesday, March 12,
2008. ERCOT is currently analyzing these issues to determine the volume
and transactions impacted. Additional information will be provided as
it becomes available.
CONTACT: If you have any questions, please contact your ERCOT Account
Manager. You may also call the general ERCOT Client Services phone
number at (512) 248-3900 or contact ERCOT Client Services via e-mail at
[log in to unmask]
If you are receiving e-mail from an ERCOT distribution list that you no
longer wish to receive, please follow this link in order to unsubscribe
from this list: http://lists.ercot.com <http://lists.ercot.com/> .
DP/JA
________________________________
From: ERCOT Client Relations
Sent: Wednesday, March 12, 2008 10:57 AM
To: [log in to unmask];
[log in to unmask];
[log in to unmask]
Cc: [log in to unmask]; [log in to unmask];
[log in to unmask]; 1 ERCOT Client Service Reps
Subject: M-A031208-01 Outage - Retail/Wholesale
NOTICE DATE: March 12, 2008
NOTICE TYPE: M-A031208-01 Outage - Retail/Wholesale
CLASSIFICATION: Public
SHORT DESCRIPTION: This morning, ERCOT experienced issues with the
Customer Registration Application servers.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: Wednesday, March 12, 2008 from 9:08 AM to 10:15 AM
LONG DESCRIPTION: This morning, Wednesday, March 12, 2008 from 9:08 AM
to 10:15 AM ERCOT experienced an issue with the Customer Registration
Application servers. Impacts included Retail Transaction Processing,
Texas Market Link Query ESIID Transaction, eServices Disputes and
Service Requests, and MarkeTrak.
ERCOT's initial assessment has determined that the root cause is due to
a hardware failure. Additional details will be provided in the ERCOT IT
Incident Market Notice Summary.
CONTACT: If you have any questions, please contact your ERCOT Account
Manager. You may also call the general ERCOT Client Services phone
number at (512) 248-3900 or contact ERCOT Client Services via e-mail at
[log in to unmask]
If you are receiving e-mail from an ERCOT distribution list that you no
longer wish to receive, please follow this link in order to unsubscribe
from this list: http://lists.ercot.com <http://lists.ercot.com/> .
TR/JA
|
