NOTICE DATE: April 15, 2008
NOTICE TYPE: M-A031208-08 UPDATE:
Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT has
identified and reprocessed twenty-six (26) additional files.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 10, 2008 through
March 21, 2008
LONG DESCRIPTION: ERCOT has
identified and reprocessed twenty-six (26) additional 867_03 files that were
not loaded into the Settlements and Billing application during the March 10 through
March 21, 2008 timeframe when ERCOT was experiencing retail transaction
processing issues.
The
following are updated counts of impacted 867_03s:
|
Count |
Files |
NIDR |
12504 |
1 |
IDR |
431 |
57 |
ADDITIONAL INFORMATION: ERCOT
Retail Account Managers will provide all affected Market Participants with a
list of the additional impacted 867_03s.
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT
Client Services phone number at (512) 248-3900 or contact ERCOT Client Services
via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
TR/DP
From: ERCOT Client Relations
Sent: Thursday, April 10, 2008
2:52 PM
To:
Cc:
Subject: M-A031208-07 UPDATE:
Outage/Retail Processing
NOTICE DATE: April 10, 2008
NOTICE TYPE: M-A031208-07 UPDATE:
Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT has
identified and reprocessed sixteen (16) additional files.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008 through
March 21, 2008
LONG DESCRIPTION: ERCOT has identified
and reprocessed sixteen (16) additional 867_03 files that were not loaded into
the Settlements and Billing application during the March 12 through March 21,
2008 timeframe when ERCOT was experiencing retail transaction processing
issues.
ERCOT
continues working to identify any additional unprocessed files so that they may
be reprocessed. As a result of reprocessing, Market Participants may
receive duplicate 867_03 forwards. The following are updated counts of
impacted 867_03s:
|
Count |
Files |
NIDR |
12504 |
1 |
IDR |
247 |
31 |
ADDITIONAL INFORMATION: ERCOT
Retail Account Managers will provide all affected Market Participants with a
list of the additional impacted 867_03s.
CONTACT: If you have any questions, please
contact your ERCOT Account Manager. You may also call the general ERCOT Client
Services phone number at (512) 248-3900 or contact ERCOT Client Services via
e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
TR/JA
From: ERCOT Client Relations
Sent: Wednesday, April 09, 2008
1:40 PM
To:
Cc:
Subject: M-A031208-06 UPDATE:
Outage/Retail Processing
NOTICE DATE:
April 9, 2008
NOTICE TYPE: M-A031208-06 UPDATE: Outage/Retail
Processing
CLASSIFICATION: Public
SHORT DESCRIPTION:
Additional impacts due to the retail transaction processing outage that
occurred on March 12, 2008.
INTENDED AUDIENCE:
CRs and TDSPs
DAY AFFECTED:
March 12, 2008 through March 21, 2008
LONG DESCRIPTION:
ERCOT has been made aware of additional impacts due to the
retail
transaction processing outage that occurred on March 12, 2008. Some
867_03 files were not processed. The files were received by ERCOT and
acknowledged but were not included in the files created for data loading into
the Settlements and Billing application. Both IDR and NIDR files were
among those affected.
ERCOT is working to identify the
unprocessed files so that they may be reprocessed. As a result of
reprocessing, Market Participants may receive duplicate 867_03 forwards.
Following are initial counts of impacted 867_03:
|
Count |
Files |
NIDR |
12504 |
1 |
IDR |
117 |
15 |
The outages and Retail processing
issues encountered were the result of the following:
1)
Failed production release of SCR745
(Retail Market Outage Evaluation and Resolution) - Due to slow performance and
crashes, an outage outside of the standard maintenance window was utilized to
rollback to pre-implementation state.
2)
Storage hardware failures – A
project to replace aged hardware has been in progress since late 2007 and is
scheduled for implementation in May.
3)
Human error – Operating
procedure revisions have been made to prevent future occurrences as well as
education and training.
ADDITIONAL INFORMATION: ERCOT
Retail Account Manager’s will provide all affected Market
Participant’s with a list of impacted 867_03s. The lists are
expected to be ready by Thursday, April 10, 2008 after 9:00 AM.
CONTACT: If you have
any questions, please contact your ERCOT Account Manager. You may also call the
general ERCOT Client Services phone number at (512) 248-3900 or contact ERCOT
Client Services via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution
list that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/DG
From: ERCOT Client Relations
Sent: Thursday, March 20, 2008
5:42 PM
To:
Cc:
Subject: M-A031208-05 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 20, 2008
NOTICE TYPE: M-A031208-05
UPDATE: Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION:
Additional transactions have been impacted by the retail transaction
processing outage that occurred on March 12, 2008.
INTENDED AUDIENCE: CRs and
TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: The
following transactions were impacted as a result of the retail transaction
processing outage on March 12, 2008.
Details
on impacted transactions:
·
867s - ERCOT had errors during the
reprocessing efforts on March 18, 2008 and March 19, 2008. ERCOT will be
checking with the two (2) TDSPs again for approval to reprocess.
Updated counts
|
DUNS |
Count |
867s |
10994067400 |
10,008 |
867s |
957877905 |
106,477 |
·
ERCOT analysis has identified an
additional set of 867 forwarding delays.
Updated counts
|
DUNS |
Count |
867 forwards |
1039940674000 |
19628 |
·
ERCOT Retail Account
Manager’s will provide all affected Market Participant’s with a
list of impacted 867 transactions (both the reprocessing and the forwarding
delays) after 8:00 AM on Friday, March 21, 2008.
·
814s - ERCOT has completed the
ERCOT Initiated MarkeTrak issues for 814s that have been identified as not
processing to request approval to reprocess. The final count of impacted
transactions was 139.
·
ERCOT Retail Account
Manager’s have been provided a list of MarkeTrak issue numbers and will
provide those to the affected Market Participants.
·
997s - ERCOT is reprocessing
current missing 997 requests via MarkeTrak issues and performing additional
analysis for any potential impacts.
ADDITIONAL INFORATION: If Market
Participants are missing transactions, ERCOT requests that a MarkeTrak issue be
filed, with an attached spreadsheet, for ERCOT to research.
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT
Client Services phone number at (512) 248-3900 or contact ERCOT Client Services
via e-mail at [log in to unmask].
If you are receiving
e-mail from an ERCOT distribution list that you no longer wish to receive,
please follow this link in order to unsubscribe from this list: http://lists.ercot.com.
JA/DG
From: ERCOT Client Relations
Sent: Tuesday, March 18, 2008 5:21
PM
To:
Cc:
Subject: M-A031208-04 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 18, 2008
NOTICE TYPE: M-A031208-04
UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT has
determined that a subset of transactions on March 12, 2008 was not processed
through ERCOT systems.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT has
determined that the retail transaction processing issues, which occurred on
March 12, 2008, were the result of human error. A slow-down on the
application server was being investigated and was believed to be a hardware
issue. During the investigation, the batch process was mistakenly restarted
while one was already running. This caused errors with the transaction
processing and corrupted log files.
Reason for delay in communication to the Market: The
validity of the exception queries that run against the logs was in
question. The effort continues to be a challenge due to the volume of
data and the anomaly of the problem that occurred with log files being
corrupted or missing as a result.
ADDITIONAL INFORMATION: Details on
impacted transactions:
867s
- ERCOT Initiated MarkeTrak issues were submitted to two (2) TDSPs and ERCOT received the approval to
reprocess (on March 18, 2008). ERCOT has begun to reprocess and forward
the 867s.
|
DUNS |
Count |
867s |
1039940674000 |
10,008 |
867s |
957877905 |
106,477 |
ERCOT
will continue analysis to identify any additional 867 forwarding delays.
814s
- ERCOT is logging ERCOT Initiated MarkeTrak issues for 814s that have been
identified as not processing to request approval to reprocess. The
estimated count is ~ 150 transactions.
997s
- ERCOT is reprocessing current missing 997s requests via MarkeTrak issues and
performing additional analysis for any potential impacts.
ERCOT
Retail Account Manager’s will provide all affected Market
Participant’s with a list of impacted 867 transactions after 2:00 PM on
Wednesday, March 19, 2008. 814s
will be communicated as ERCOT initiated MarkeTrak issues.
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT Client
Services phone number at (512) 248-3900 or contact ERCOT Client Services via
e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/TR
From: ERCOT Client Relations
Sent: Monday, March 17, 2008 3:58
PM
To:
Cc:
Subject: M-A031208-03 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 17, 2008
NOTICE TYPE: M-A031208-03
UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT experienced retail
transaction processing issues on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT continues to
analyze these processing issues to determine the volume and transactions impacted.
Additional
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT
Client Services phone number at (512) 248-3900 or contact ERCOT Client Services
via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/JA
From: ERCOT Client Relations
Sent: Friday, March 14, 2008 4:39
PM
To:
Cc:
Subject: M-A031208-02 UPDATE:
Outage/Retail Processing
NOTICE DATE: March 14, 2008
NOTICE TYPE: M-A031208-02
UPDATE - Outage/Retail Processing
CLASSIFICATION: Public
SHORT DESCRIPTION: ERCOT experienced
retail transaction processing issues on March 12, 2008.
INTENDED AUDIENCE: CRs and TDSPs
DAY AFFECTED: March 12, 2008
LONG DESCRIPTION: ERCOT has
discovered an issue processing a subset of retail transactions as a result of
the outage on Wednesday, March 12, 2008. ERCOT is currently analyzing
these issues to determine the volume and transactions impacted.
Additional
CONTACT: If you have any questions,
please contact your ERCOT Account Manager. You may also call the general ERCOT
Client Services phone number at (512) 248-3900 or contact ERCOT Client Services
via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution list
that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
DP/JA
From: ERCOT Client Relations
Sent: Wednesday, March 12, 2008
10:57 AM
To:
Cc:
Subject: M-A031208-01 Outage
– Retail/Wholesale
NOTICE DATE: March
12, 2008
NOTICE TYPE: M-A031208-01
Outage – Retail/Wholesale
CLASSIFICATION: Public
SHORT DESCRIPTION:
This morning, ERCOT experienced issues with the Customer Registration
Application servers.
INTENDED AUDIENCE: CRs
and TDSPs
DAY AFFECTED:
Wednesday, March 12, 2008 from 9:08 AM to 10:15 AM
LONG DESCRIPTION:
This morning, Wednesday, March 12, 2008 from 9:08 AM to 10:15 AM
ERCOT
experienced an issue with the Customer Registration Application servers.
Impacts included Retail Transaction Processing, Texas Market Link Query ESIID
Transaction, eServices Disputes and Service Requests, and MarkeTrak.
ERCOT’s initial assessment
has determined that the root cause is due to a hardware failure.
Additional details will be provided in the ERCOT IT Incident Market Notice
Summary.
CONTACT: If you have
any questions, please contact your ERCOT Account Manager. You may also call the
general ERCOT Client Services phone number at (512) 248-3900 or contact ERCOT
Client Services via e-mail at [log in to unmask].
If you are receiving e-mail from an ERCOT distribution
list that you no longer wish to receive, please follow this link in order to
unsubscribe from this list: http://lists.ercot.com.
TR/JA