QB / QB64 Discussion Forum      Other Subforums, Links and Downloads
 


  << Previous Topic | Next Topic >>Return to Index  

Rounding Numbers and Round-off Errors

December 8 2002 at 12:13 PM
Solitaire  (no login)

 
Some time ago I discovered that QBasic will round out numbers ending with .5 to the nearest EVEN integer, or any fractional number ending with 5 to the nearest EVEN least significant digit up to the limit of the type range. For example, if you declare a number AS INTEGER and assign it the value of 3.5, it will become 4, and if you assign it the value of 4.5, it will also become 4. Similarly, if you assign a Single (or default type) number the value of .12342335 it will print as .1234234 and .12342345 will also print as .1234234. The same will happen with default Single numbers 1234.2335 and 1234.2345. The CINT() function will also round out to the nearest even integer. Try this:

CLS
DIM num1 AS INTEGER, num2 AS INTEGER
num1 = 3.5
num2 = 4.5
num3 = .12342335#
num4 = .12342345#
num5 = 1234.2335#
num6 = 1234.2345#
num7 = 3.5
num8 = 4.5
PRINT num1
PRINT num2
PRINT num3
PRINT num4
PRINT num5
PRINT num6
PRINT CINT(num7)
PRINT CINT(num8)
END


The same is true for the Round() function in Visual Basic 6. I was wondering if this was some kind of bug, until I checked the Internet and searched for Rounding Numbers. There seems to be a difference of opinion on how numbers ending with .5 are to be rounded. Most of us would round UP to the next highest number, as we were taught in mathematics classes. However, the preferred way is to round to the nearest EVEN number! I suppose this is used as the preferred method in order to avoid significant round-off errors. Look at it this way: Groups of numbers ending with 1 - 4 and groups of numbers ending with 6 - 9 would statistically be about the same. That leaves the odd group of numbers ending with the digit 5. If they were all to be rounded UP, then eventually, if thousands of numbers are totalled, The result would be too big, producing a large round-off error. On the other hand, if half of the numbers ending with 5 were rounded down and half rounded up, the error would not be as great. With individual or small groups of numbers, however, you would probably prefer to use the always round UP method.

I checked the Excel spreadsheet program to see how the rounding function worked there, and to my surprise, it always rounded UP if a number ended with 5. I decided to write a program with a function that will round out any number, positive or negative to any place selected by the user, using the round UP method same as in Excel. If the number is too big or too small for the largest data type, it will produce an error, so I included an errortrap to end the function if that happens. The code for this program appears below.

You should be aware that a round-off error may still occur if the results you require use the same number of places after the decimal as the number of places you have rounded out. Knowing how to round out a number is not the same as solving a round-off error; just the opposite. Eventually the missing fraction lost in multiple roundouts compounds the error and produces a result which is no longer accurate to the required degree. To avoid round-off errors, you must attempt to extend the number of decimal places as far as possible before rounding out.

The problem is that the computer is a finite machine and can only produce results to a finite degree of accuracy. After a certain number of decimal places have been reached within the memory limits of the data type, the computer rounds out the last digit automatically. Using a data type with the greatest range from minimum to maximum helps extend the accuracy. The SINGLE data type is accurate only to 7 significant digits. The DOUBLE data type is accurate to 15 significant digits. The digit which would follow that number has been rounded out. If you want to use a number with greater accuracy, the newer version of Visual Basic includes a Decimal type, which is accurate to 28 digits. But some numbers are infinite, and no matter how powerful a computer you use, even if you use a super-computer which can handle numbers up to a couple of hundred places, the last place will still have to be rounded out.

The point is, you have to know what your requirements are, and how many significant digits are needed in order to produce the desired result without incurring a round-off error. If it's greater than the number available with QBasic's data types, then it simply will not produce accurate results. An undeclared variable in QBasic is SINGLE by default. If you have not declared your variables by type, then you can increase the accuracy of your results by declaring your number variable as DOUBLE. Put this statement at the beginning of your program, using the name of your variable for the number. Also see my code for the RoundUP function, which follows.

DIM number AS DOUBLE

--------------------------------------------------

DECLARE FUNCTION RoundUp# (num AS DOUBLE, place AS INTEGER)
DIM num AS DOUBLE, place AS INTEGER, splace AS STRING, E AS STRING
DO
CLS
PRINT "Program will round to nearest digit for specified number ";
PRINT "of decimal places,"
PRINT "or will round to least significant digit of an integer."
PRINT "It will work with either a positive or negative number."
PRINT "Zero places will round to nearest integer."
PRINT : INPUT "Enter number to round: ", num
PRINT : PRINT "Enter number of places to round after the decimal,"
PRINT "negative number of places to left of decimal, or 0 for "
INPUT "rounded integer (place must be between -9 and 9.): ", splace$
place = VAL(splace)
IF place >= -9 AND place <= 9 THEN
PRINT : PRINT "Rounded number is "; RoundUp(num, place)
ELSE
PRINT "Out of range."
END IF
PRINT : INPUT "Enter for another number or Q to quit: ", E$
LOOP UNTIL UCASE$(E$) = "Q"
END

FUNCTION RoundUp# (num AS DOUBLE, place AS INTEGER)
DIM decnum AS DOUBLE, decplace AS LONG
IF place > 9 OR place < -9 THEN EXIT FUNCTION 'otherwise incorrect
IF place >= 0 THEN
decplace = 10 ^ place
decnum = num * decplace
IF decnum > 0 THEN
decnum = INT(decnum + .5)
ELSE
decnum = FIX(decnum - .5)
END IF
RoundUp = decnum / decplace
ELSEIF place < 0 THEN
place = ABS(place)
decplace = 10 ^ place
decnum = num / decplace
IF decnum > 0 THEN
decnum = INT(decnum + .5)
ELSE
decnum = FIX(decnum - .5)
END IF
RoundUp = decnum * decplace
END IF
END FUNCTION

 
 Respond to this message   
AuthorReply
Earthborn
(no login)

Very nice! But could be shortened though...

December 10 2002, 1:02 PM 

How about this:

FUNCTION Round (Num, Decimals)
n = Num * (10 ^ Decimals)
n = SGN(n) * (ABS(INT(n + .5)))
Round = n / (10 ^ Decimals)
END FUNCTION

 
 Respond to this message   
Solitaire
(no login)

*Excellent! Thank you.

December 10 2002, 1:47 PM 


 
 Respond to this message   
Moneo
(no login)

BEWARE! Earthborn's function looked nice until I tested it >>>

November 7 2007, 6:21 PM 

The following is Earthborn's original function:

FUNCTION Round (Num, Decimals)
n = Num * (10 ^ Decimals)
n = SGN(n) * (ABS(INT(n + .5)))
Round = n / (10 ^ Decimals)
END FUNCTION

The error lies in this statement:
n = SGN(n) * (ABS(INT(n + .5)))

The intent of the ABS is to feed an absolute value to the INT, because INT works differently for negative numbers.

The corrected function follows:

FUNCTION Round (Num, Decimals)
n = Num * (10 ^ Decimals)
R = SGN(N)*INT(ABS(N)+.5) '*** FIXED ***
Round = n / (10 ^ Decimals)
END FUNCTION

NOTE: Earthborn's function did not declare the rounding method used. It is Symmetric Arithmetic Rounding, the most commonly used method.

Regards..... Moneo







 
 Respond to this message   

(Login burger2227)
R

Really?

November 12 2007, 1:55 PM 

INT(4.1) = 4
INT(4.9) = 4
INT(-4.1) = -5
INT(-4.9) = -5

I think your forgetting that -5 is lower than -4. INT always returns a lower value until it increases to the next integer value, which is one higher.

ABS is used to keep the resulting number positive.

FIX will raise the value one on negative numbers and is the same as INT on positive numbers. Try that if you feel the need.

Ted

PS: You waited 5 years to unearth this startling revelation? I will notify Solitaire who probably has been happily using it all of that time.

 
 Respond to this message   
Moneo
(no login)

Ted, don't be so critical. Listen to the following:

November 12 2007, 5:55 PM 

How the INT function works for positive and negative numbers, I am well aware.

The point of the post that I started was the testing of Earthborn's function. He didn't say, but from looking at the code one must assume that the rounding being done is the standard Symmetric Arithmetic Rounding (SAR) method.

Testing his fonction as is, you will get:
-1.5 rounds to -1, where the SAR method gives -2.
-2.5 rounds to -2, where the SAR method gives -3.

It's remotely possible that he was implementing some other rounding method, but if so, he certainly should have declared which method.

My correction to his code will perform rounding according to the SAR method.

Regards..... Moneo


 
 Respond to this message   

(Login burger2227)
R

OK, explain this then

November 12 2007, 7:31 PM 

I took Earthborn's and your functions and made two of my own:

DECLARE FUNCTION RMoneo! (Num!, Decimals!)
DECLARE FUNCTION Round! (Num!, Decimals!)
DECLARE FUNCTION RTed! (Num!, Decimals!)
INPUT "Enter a number: ", Num
Decimals = 2



COLOR 12: PRINT Round(Num, Decimals)

COLOR 14: PRINT RMoneo(Num, Decimals)

COLOR 10: PRINT RTed(Num, Decimals)

FUNCTION RMoneo (Num, Decimals)
n = Num * (10 ^ Decimals)
n = SGN(n) * INT(ABS(n) + .5)'*** FIXED *** I changed R to n
RMoneo = n / (10 ^ Decimals)
END FUNCTION

FUNCTION Round (Num, Decimals)
n = Num * (10 ^ Decimals)
n = SGN(n) * (ABS(INT(n + .5)))
Round = n / (10 ^ Decimals)
END FUNCTION

FUNCTION RTed (Num, Decimals)
n = Num * (10 ^ Decimals)
n = INT(n + .5)
RTed = n / (10 ^ Decimals)
END FUNCTION

FUNCTION RFix (Num, Decimals)
n = Num * (10 ^ Decimals)
n = FIX(n + .5)
RFix = n / (10 ^ Decimals)
END FUNCTION


I tried all four and Earthborn's and mine were identical even with negative numbers. With just 2 decimal places I tried 11.999 and our's rounded to 12 while yours stayed 11.999. Then I noticed an R where n should be in your post. Now they seem to be almost the same, but yours goes lower sometimes with negative numbers. I don't think that was the original intent of Earthborn's post however.

You will notice that I removed all refernces to SGN(n) and ABS. The sign came up correctly on mine despite that. So who really needs them? SGN saves the number's sign which is NEVER multiplied by a negative number in ANY of the functions. Thus ABS(n) CREATES the sign problem to begin with! Your's works for what you desired I imagine. So how can you call his formula any worse than our's? Solitaire liked it that way. I also made a function with FIX to see if that made a difference. Your's was the only one different!

**********************************************
Rounding Down as explained by Microsoft

The simplest form of rounding is truncation. Any digits after the desired precision are simply ignored. The Fix() function is an example of truncation. For example, Fix(3.5) is 3, and Fix(-3.5) is -3.

The Int() function rounds down to the highest integer less than the value. Both Int() and Fix() act the same way with positive numbers - truncating - but give different results for negative numbers: Int(-3.5) gives -4.

>>>> The Fix() function is an example of symmetric rounding because it affects the magnitude (absolute value) of positive and negative numbers in the same way. The Int() function is an example of asymmetric rounding because it affects the magnitude of positive and negative numbers differently.
***********************************************
Since your rounding actually increases the negative value, it cannot be Symmetrical rounding!

Obviously you never used the IDE, because there are N's and n's and that strange R. Use the IDE friend. I copy my code straight from a module using Notepad AFTER I run it in the IDE.

Regards, Ted

 
 Respond to this message   
Moneo
(no login)

Ted, thanks for fixing my coding error >>>

November 13 2007, 3:30 PM 

I lifted the code from a working program of mine, and didn't test it again because it was already working. "Famous last words." The variable names were different. Sorry.

In the Microsoft document
http://support.microsoft.com/kb/196652/
that you quoted from, there's a table further down with the heading of "Sample Data". In this table you will notice that the column for Asymmetric Arithmetic and the column for Symmetric Arithmetic demonstrate the same differences between Earthborn/your rounding functions and my rounding function. The differences are:
For -2.5, yours gives -2, and mine -3
For -1.5, yours gives -1, and mone -2
For 0.5, yours gives 0, and mine -1.

My algorithm performs Symmetric Arithmetic Rounding, and the SGN and ABS are absolutely necessary. I don't understand why you don't accept this. This Symmetric Arithmetic algorithm has been discussed over the last 5 years on several QB forums as well as on the MathForum.

From the results that I see for Earthborn and your algoritms, it seems like they are performing Asymmetric Arithmetic Rounding. Having never worked with this rounding method, I can't say for sure.

We still don't know if Earthborn was implementing Symmetric of Asymmetric Arithmetic Rounding. Testing shows that the results will be the same for both methods, except for negative numbers that end in .5. If he or Solitaire never tested with this combination of values, their tests would have been satisfactory for them.

Thanks for all the work and effort that you put into this issue.

You're probably right about the IDE, but, "you can't teach an old dog new tricks."

Regards..... Moneo



 
 Respond to this message   

(Login burger2227)
R

Dear OLD Dog ..........................

November 13 2007, 8:36 PM 

If you look at the code, you will notice that Earthborn added .5 to the number, no matter what value it was. That tells me that he was rounding up. If your rounding method rounds up then your theory is perfectly Symmetric.

However, the rounding done, with all of the ABS, SGN and INT stuff still rounds lower than what it could have been. Your negatives increase, the negative numbers and just do not make much sense to me, but who am I to tell you it is wrong?

I know one thing, you will NEVER get any money from me to invest. If I am losing money and round off like you, I will be double penalized!

Ted

PS: You just chastized me for picking on Antoni and you post BEWARE? Like it will mess up your PC? Common, buddy you should have just said something like "Symmetrical Rounding".

 
 Respond to this message   
Moneo
(no login)

Symmetric Arithmetic Rounding accoroding to Microsoft document

November 14 2007, 9:08 AM 

Ted,
Here's an extract from the Arithmetic Rounding section of the Microsoft document on rounding.

"However, what about 1.5, which is equidistant between 1 and 2? By convention, the half-way number is rounded up.
You can implement rounding half-way numbers in a symmetric fashion, such that -.5 is rounded down to -1, or in an asymmetric fashion, where -.5 is rounded up to 0."

Notice that for Symmetric Arithmetic Rounding,
-.5 is rounded to -1.
and therefore implies that:
-1.5 is rounded to -2
-2.5 is rounded to -3.

Now, if Symmetric Arithmetic Rounding does not meet with your rounding requirements, you can chose to use some other rounding method. See the "Sample Data" table in the Microsoft document for examples of how other rounding methods perform as opposed to Symmetric Arithmetic Rounding.

Regards..... Moneo




 
 Respond to this message   

(Login burger2227)
R

* The point here is that both rounding methods are fine!

November 14 2007, 3:48 PM 


 
 Respond to this message   
Moneo
(no login)

* Great! We agree on something.

November 14 2007, 7:56 PM 

*

 
 Respond to this message   

(Login burger2227)
R

* In a Round about way LOL

November 14 2007, 8:38 PM 


 
 Respond to this message   
Solitaire
(Login Solitaire1)
S

Information about arithmetic rounding vs. banker's rounding

September 6 2004, 5:21 PM 

This is a Microsoft Knowledge Base Article:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;196652


It's not specific to QB but it does refer to several functions that are also available in QB and work the same way as described.


 
 Respond to this message   

(Login Mac36)

Is there a better way to simulate CINT?

October 22 2004, 11:48 AM 

We know that CINT(x) will fail unless x is integer size.

Here is a function that will handle larger values. Can you find a better way?

(Just replace the code of CDBLx# with your better way and post)

Mac

DECLARE FUNCTION CDBLx# (k#)
CLS
PRINT "Proof the function works"
k$ = "34.4": GOSUB Testit
k$ = "34.5": GOSUB Testit
k$ = "34.6": GOSUB Testit
k$ = "35.4": GOSUB Testit
k$ = "35.5": GOSUB Testit
k$ = "35.6": GOSUB Testit
SYSTEM

Testit:
PRINT k$; "  "; CDBLx#(VAL(k$)), CDBLx#(VAL("888888" + k$))
PRINT k$; "  "; CDBLx#(VAL("-" + k$)), CDBLx#(VAL("-888888" + k$))
RETURN

FUNCTION CDBLx# (k#)
Max% = 32767
IF ABS(k#) < Max% THEN
  r# = CINT(k#)
ELSE
  'Need to simulate CINT
  k$ = STR$(k#)
  y = INSTR(k$, ".")
  IF y = 0 THEN
    r# = k#
  ELSE
    k1$ = LEFT$(k$, y - 1): k2$ = RIGHT$(k$, LEN(k$) - y + 1)
    r# = VAL(k1$)
    IF r# < 0 THEN
      IF k2$ = ".5" THEN
        IF INSTR("02468", RIGHT$(k1$, 1)) = 0 THEN r# = r# - 1
      ELSE
        IF VAL(k2$) > .5 THEN r# = r# - 1
      END IF
    ELSE
      IF k2$ = ".5" THEN
        IF INSTR("02468", RIGHT$(k1$, 1)) = 0 THEN r# = r# + 1
      ELSE
        IF VAL(k2$) > .5 THEN r# = r# + 1
      END IF
    END IF
  END IF
END IF
CDBLx# = r#
END FUNCTION


 
 Respond to this message   

(no login)

Re: Is there a better way to simulate CINT?

October 23 2004, 4:05 PM 

Amount#=12345.789123
Dollar&=Amount#
Cent&=(Amount#-Dollar&)*100

print using "##,###,###.##";Dollar&+Cent&/100

 
 Respond to this message   

(no login)

Re: Is there a better way to simulate CINT?

October 23 2004, 4:20 PM 

BTW if you want to store it compressed in a file

Amount#=12345.678912
Dollar&=Amount#
Cent&=(Amount#-Dollar&)*100
if Cent&<0 then
Dollar&=Dollar&-1
Cent&=Cent&+100
end if
st$=MKL$(Dollar&)+CHR$(Cent&)

makes a string 5 characters containing up to
+-2.1 billion +2 digit decimal.

then unpack with:

Dollar&=CVL(mid$(st$,1,4))
Cent&=Asc(mid$(st$,5,1) - or just mid$(st$,5)

print Dollar&+Cent&/100


 
 Respond to this message   

(Login Mac36)

Hey, Buff1 (and Solitaire)

October 24 2004, 6:14 PM 

How about taking my original program and throwing out the stuff between
FUNCTION CDBLx# (k#)
and
END FUNCTION

Then provide your better algorithm. Debug it and post. That way it could be tested and compared.

Thanks

Mac



 
 Respond to this message   
Solitaire
(Login Solitaire1)
S

My version of banker's rounding with large numbers using CINT

October 24 2004, 2:46 PM 

I didn't look at Mac's code. This was done from scratch. I used string manipulation to separate the ones place digit and fractional part from the rest of the number. Converted and rounded the small number with CINT. Concatenated a "0" to the rest of the number before converting it back to a double, then added it to the small rounded number in the ones place.

Basically, since banker's rounding only operates on the digit in the one's place (using the digits to the right of the decimal as the argument), that is all that's really needed, and CINT is adequate for that purpose. The remaining digits to the left of the one's place stay the same no matter how big the number is.

I did this in a hurry but I'm sure it can be converted into a function. First, test it to make sure there's no error.


DIM newnum AS DOUBLE, pnum AS DOUBLE, rnum AS SINGLE
DIM dot AS INTEGER, inum AS INTEGER
DIM snum AS STRING, cnum AS STRING, lnum AS STRING
CLS : INPUT "Enter a number to round:  ", snum$
dot = INSTR(snum$, ".")
IF dot = 0 OR dot = 1 THEN
    cnum$ = snum$       'ones digit + fraction
ELSE
    cnum$ = MID$(snum$, dot - 1)
    lnum$ = LEFT$(snum$, dot - 2)   'left digits up to tens
END IF
IF dot > 0 THEN
    rnum = VAL(cnum$)       'convert ones & fraction to number
    inum = CINT(rnum)       'rounded and changed to integer
    lnum$ = RTRIM$(lnum$) + "0" 'put 0 back in ones place for whole number
    pnum# = VAL(lnum$)       'convert whole number
    newnum# = pnum + inum    'add whole number to rounded ones place
ELSE
    newnum# = VAL(snum$)
END IF
PRINT "Rounded whole number is  "; newnum#
SYSTEM


 
 Respond to this message   
Solitaire
(no login)

My banker's rounding function integrated with Mac's main program

October 25 2004, 11:06 PM 

DECLARE FUNCTION CDBLx# (k#)
CLS
PRINT "Proof the function works"
k$ = "34.4": GOSUB Testit
k$ = "34.5": GOSUB Testit
k$ = "34.6": GOSUB Testit
k$ = "35.4": GOSUB Testit
k$ = "35.5": GOSUB Testit
k$ = "35.6": GOSUB Testit
SYSTEM

Testit:
PRINT k$; "  "; CDBLx#(VAL(k$)), CDBLx#(VAL("888888" + k$))
PRINT k$; "  "; CDBLx#(VAL("-" + k$)), CDBLx#(VAL("-888888" + k$))
RETURN

FUNCTION CDBLx# (k#)
DIM pnum AS DOUBLE, rnum AS SINGLE
DIM dot AS INTEGER, inum AS INTEGER
DIM snum AS STRING, cnum AS STRING, lnum AS STRING

snum$ = STR$(k#)
dot = INSTR(snum$, ".")
IF dot = 0 OR dot = 1 THEN
    cnum$ = snum$                   'ones digit + fraction
ELSE
    cnum$ = MID$(snum$, dot - 1)
    lnum$ = LEFT$(snum$, dot - 2)   'left digits up to tens place
END IF
IF dot > 0 THEN
    rnum = VAL(cnum$)               'convert ones & fraction to number
    IF k# < 0 THEN rnum = -rnum     'negative number
    inum = CINT(rnum)               'rounded and changed to integer
    lnum$ = RTRIM$(lnum$) + "0"     'put 0 back in ones place for whole number
    pnum# = VAL(lnum$)              'convert whole number
    CDBLx# = pnum + inum            'add whole number to rounded ones place
ELSE
    CDBLx# = num#                   'no fraction given to round
END IF
END FUNCTION


 
 Respond to this message   

(Premier Login iorr5t)
Forum Owner

Hmmm, What about this rounding function?

October 27 2004, 5:11 AM 

The theory: on numbers greater than 2,147,483,647 it doesn't matter what rounding is used. Might as well use FIX. Nobody knows anything to that much accuracy.

Mac

FUNCTION CDBLx# (k#)
CONST MaxINT = 32767
CONST MaxLNG = 2147483647
IF ABS(k#) <= MaxINT THEN
  CDBLx# = CINT(k#)
ELSE
  IF ABS(k#) <= MaxLNG THEN CDBLx# = CLNG(k#) ELSE CDBLx# = FIX(k#)
END IF
END FUNCTION

 
 Respond to this message   
Moneo
(no login)

Simulate CINT using an "integer divide"

October 16 2007, 2:49 PM 

As Mac indicated, the problem with CINT is that it can't handle numbers outside the range of an integer, i.e., -32768 to 32767. Therefore, for these large numbers, Mac was looking for a method which simulated CINT.

An integer divide of the number using a divisor of 1, will perform the same operation as a CINT, but without the integer range restriction.

DIM NUMBER AS SINGLE
DIM RESULT AS SINGLE

RESULT = CINT(NUMBER) 'Issues error if NUMBER out of integer range.
'Instead do this:
RESULT = NUMBER\1

It turns out that CINT, INTEGER ASSIGNMENT, and INTEGER DIVISION all perform Banker's Rounding when rounding.

CINT preforms Banker's Rounding, but the result is restricted to numbers within integer range. The variable receiving the rounded result can be an integer, long or double.

INTEGER ASSIGNMENT is similar to CINT, and also performs Banker's Rounding. By definition, the variable receiving the rounded result must be an integer.

INTEGER DIVISION does a 3 step operation.
1) It rounds the dividend using Banker's Rounding.
2) It rounds the divisor using Banker's Rounding.
3) Divides the dividend by the divisor, and then TRUNCATES the result.

Regards..... Moneo

 
 Respond to this message   

(Login MystikShadows)
R

Hmmm, doesn't QB, QBasic, PDS and VB-DOS have something for that?

October 16 2007, 4:07 PM 

What about the CLNG function? Ir is the goal here to actually simulate it? :-) just making sure, I think CLNG is in the help file of probably all these interpreters/compilers :-).

None the less, it's very interesting to see how it works the way it's implemented.


 
 Respond to this message   
Solitaire
(Login Solitaire1)
S

* CLNG does the same rounding as CINT but for a greater range.

October 16 2007, 4:52 PM 


 
 Respond to this message   
Moneo
(no login)

* You're right, Solitaire. Never knew CLNG existed.

October 16 2007, 5:51 PM 


 
 Respond to this message   
Moneo
(no login)

Weirdness found using CINT or CLNG

October 16 2007, 6:20 PM 

I was testing CINT, CLNG, integer division and integer assignment, all of which use Banker's Rounding, and discovered this weird result using all of them.

DIM N AS SINGLE
DIM R AS SINGLE

R = CINT(N)

'When N = 254.50001, The result is 255 (it rounded)
'When N = 256.50001, the result is 256 (it didn't round)

Further testing revealed that when the whole part of the number is even, like those above, and this number is less than or equal to 254, with the decimal part being .50001, then the number is rounded up. For numbers equal to or greater that 256, the number does not get rounded.

We can blame it on floating point, but it's still very weird.

The CINT test above was also performed using CLNG, integer division, and integer assignment. They all behaved the same.

Regards..... Moneo




 
 Respond to this message   

(Login burger2227)
R

Why are you dimming DOUBLE numbers as SINGLE?

October 17 2007, 8:42 PM 

When I tried to enter your N values, QB IDE added # to the end, but there was no error at runtime.

The N value at least must be DOUBLE or you get 256. I tried it both ways.

In the Double value to a Single value, the one is left out of the number too.

I would not blame the results on the functions,

Ted

 
 Respond to this message   

(Premier Login iorr5t)
Forum Owner

Good catch! What say, Moneo?

October 18 2007, 9:21 AM 

DIM n AS SINGLE
n = 254.50001#
PRINT n

gives the result " 254.5".

So R = CINT(n) is the same as R = CINT(254.5)

???

Mac

 
 Respond to this message   
Moneo
(no login)

Re: Good catch! What say, Moneo?

October 18 2007, 5:37 PM 

Right, N was input as 254.50001,
but it was coerced into a SINGLE which resulted in 254.5.

Do you know a way of detecting this? Otherwise, anytime your program gets input into a SINGLE variable, the user can put in values which will not be processed as intended, because they may be coerced.

Funny this issue has not come up before, at least not for me.

All I can think of is to input the value into a string, then perform validation on the number of decimal places allowed, and the maximum whole values allowed.

Regards..... Moneo



 
 Respond to this message   
Moneo
(no login)

Re: Why are you dimming DOUBLE numbers as SINGLE?

October 18 2007, 5:21 PM 

Thanks, Ted.

You said "Why are you dimming DOUBLE numbers as SINGLE."
At first glance, how do you know that 304.50001 needs a DOUBLE variable?

This was just a little program to test the CINT function. I chose SINGLE because I intended using small numbers. The data values were keyed in by the user.

If dimming DOUBLE is the solution, why not always dim everything as DOUBLE if your program has enough memory?

If I had the input variable N as a SINGLE, how could the program determine that when N=304.50001 it should issue an error? Mac's "Edit" program, I'm sure, could figure it out, but seems like a lot of code for this problem.

Regards..... Moneo


 
 Respond to this message   

(Login burger2227)
R

The # at the end of the number was a dead giveaway!

October 18 2007, 10:39 PM 

My version of QB 4.5 placed it there when I typed it in.

I gather your's did not. So I tried making N AS DOUBLE and the .5000001 tipped the scales in favor of CINT. Of course with larger numbers CINT would bomb!

I was stumped for a while, like you were about the 255 value being rounded.
I was thinking perhaps it had something to do with 256 values in QB.

Then I came up with the fact that SINGLE cannot use that many places. So the one was never read.

Ted

 
 Respond to this message   
Moneo
(no login)

I always use QuickBasic 4.5 compiled >>>

October 19 2007, 12:24 PM 

therefore, I can't perceive the # that your interpreted QBasic version gives you automatically. Now I understand why the need for using a DOUBLE versus a SINGLE for the value in question, is so obvious to you.

What amazes me is that this problem has not become apparent on QB4.5 compiled programs that use SINGLE variable types. What I now realize that's happening, is that when the user enters a value which exceeds the limits of a SINGLE. QuickBasic compiled just cavalierly truncates the input number so that it fits into the specified SINGLE variable.

The condition could be resolved by using Mac's "Edit" program to validate the input value based on what type of variable will be used to store it. The version of this "Edit" program that I have is about 220 lines of code, and therefore a heavy addition to any program that might have this problem.

Can you think of some other solution for compiled programs using SINGLE variables?

Only using DOUBLE variables has a similar problem. The user input value could conceivably go beyond the limits of a DOUBLE, and be truncated.

Regards..... Moneo

 
 Respond to this message   

(Login burger2227)
R

What do you mean by compiled?

October 19 2007, 9:41 PM 

When I MAKE the code QB4.5 does that automatically in the IDE. How do you write your programs? You have to create them before they are compiled! What is a # in an interpreted QB? Hope it's not an insult LOL.

As for figuring if you need Single or Double, use a string input and check the length of the string. Also make sure they are all number characters using MID$ and ASC(stringnumber). You could use an INKEY$ entry and not allow non-numerical key entries.

Then convert with VAL and use the proper functions for the number of digits in your program's entry. If they exceed the limit, make the user start over.
If it is not a big deal, then get a number entry and an error perhaps.

Ted

 
 Respond to this message   
Moneo
(no login)

Re: What do you mean by compiled?

October 20 2007, 7:00 PM 

I don't use any IDE to write my programs. I never liked any IDE. I use EDIT, Notepad, or a legacy editor that I have.

You're right. You would have to input the user's value into a string and then validate it. However, as we saw for the number ending in .50001, You still need some pretty fancy logic to determine that the number will exceed a SINGLE type variable.

If the input specifications for the program say, for example, that the input can only have up to 2 decimals, then yes, the input validation is relatively simple.

Thanks and regards..... Moneo


 
 Respond to this message   

(Login burger2227)
R

Singles can only use 7 numbers in length

October 21 2007, 7:42 AM 

All you would have to do is validate a decimal point with INSTR and allow 8 in that case.

I don't understand your aversion to the IDE. I do use notepad sometimes. especially with stuff off the forums. But when you test run a program, you end up in the IDE anyhow and you could have noticed the # on the end of the number then. The IDE can be a pain sometimes, but it also can help with silly mistakes and typo's too.

Ted

 
 Respond to this message   
Moneo
(no login)

Re: Singles can only use 7 numbers in length

October 21 2007, 6:18 PM 

Hey Ted, thanks for sticking with me on this.

>All you would have to do is validate a decimal point with INSTR and allow >8 in that case.

Actually the "7 numbers in length" means that you can't have more than 7 consecutive, non-zero digits. For example: 1234567000 would be a valid number to put into a SINGLE. Also 1000000000 and 1.234567 and .000001234567 are good for SINGLE. But, 1.0234567 would not fit in a SINGLE. So the validation is not so simple, not difficult, just a little tedious.


>I don't understand your aversion to the IDE. I do use notepad sometimes. >especially with stuff off the forums. But when you test run a program, >you end up in the IDE anyhow and you could have noticed the # on the end >of the number then. The IDE can be a pain sometimes, but it also can help >with silly mistakes and typo's too.

Maybe it is an aversion to IDE's, but I discarded using them back in 1987.
I write my code with an editor.
Then I immediately compile with QB4.5 from the MSDOS commandline.
Then I test, running the executable.

Therefore, I never touch an IDE for anything. Granted I miss out on the benefits of seeing the #, and any other good intentioned "helps" that it might give me, but I'm happy without it.

Regards..... Moneo


 
 Respond to this message   

(Login burger2227)
R

Some of your numbers are wrong

October 21 2007, 10:05 PM 

INTEGER maximum is: 32767 or -32768

LONG value has a maximum of: 2,147,487,647 or 10 digits with no decimal. Negative values are 1 extra like integers.

SINGLE's max is: 9,999,999 or 7 digits (+ or -)

DOUBLE's Max is 999,999,999,999,999 or 15 digits long (+ or -).

The following are SINGLE numbers:

1234.567 1.234567 .1234567 zeros don't count on either end, only in between numbers > 0. Decimals at the end on the right are actually integers.

Your 256.50001 was not SINGLE but DOUBLE, because the string length with the decimal point was greater than 8 characters. The count can only be 7 if there is no decimal point. I guess you would have to filter out 0's before and at the end too if the INPUT is a string or uses INKEY$.

Most people would not add those zeros in an entry however unless they were before a decimal point or actually are an integer with no decimal point. Your 1000000000 is a DOUBLE number or actually LONG INTEGER.

DOUBLE sometimes adds a small amount to INTEGERs. So If there is no decimal point then use a LONG value. IF number& > 32767 AND has no decimal point THEN use a LONG integer in the program ELSE you will be sorry. LOL

That is a lot of figuring to keep from using the IDE!

Ted

PS: We better start a new thread or we will go off the screen! LOL

 
 Respond to this message   
Moneo
(no login)

Just for the record, 1000000000 can be a SINGLE.

October 22 2007, 12:19 PM 

When you put 1000000000 into a SINGLE variable and then print it, it will displayed as 1E+09, which is a valid SINGLE. There are other valid SINGLE numbers which will be displayed in exponential form. They are still valid.

True, this 1000000000 number would best be displayed using a LONG or a DOUBLE, which in this case would not need to resort to exponential form.

Ted, you're right, I think we beat this issue to death.

Regards..... Moneo

 
 Respond to this message   

(Premier Login iorr5t)
Forum Owner

Valid, maybe, but not correct

October 22 2007, 6:00 PM 

k = 10000000000#
v = 10000000999#
PRINT k, v
IF k = v THEN PRINT "yep"

This should convince you that it is not correct to put 10000000000 in a SINGLE variable. It will generate no error, but it is incorrect logically.

In scientific terms, 1E10 means you only know 1 signficant place. It means 0.5E10 to 1.5E10.

The best one can say in a single is
1000000E4, meaning 1000005e4 to 1000015e4 (or something like that - about 7 place accuracy at most.)

Thus you cannot record 10000000000 in a single variable and conserve the accuracy. It is the exact same as 10000000999. Exactly.

Mac

 
 Respond to this message   

(Login burger2227)
R

* That's where the IDE comes in handy!

October 22 2007, 9:16 PM 


 
 Respond to this message   
Moneo
(no login)

According to the famous EDIT program, it is both valid and correct

October 23 2007, 12:17 PM 

Mac, sorry for the delay. I've been doing testing with these numbers.

k = 10000000000#
v = 10000000999#
PRINT k, v
IF k = v THEN PRINT "yep"

THE ABOVE WILL PRINT 1E+10 FOR BOTH k AND v,
BUT INTERNALLY THEY ARE DIFFERENT, THEREFORE THE "yep" MESSAGE IS NOT PRINTED.

I RAN BOTH VALUES THROUGH MY VERSION OF YOUR "EDIT" PROGRAM.
10000000000 RESULTED IN AN OK SINGLE AMOUNT, DISPLAYING 1E+10.
10000000999 RESULTED IN A SINGLE W/ACCURACY LOSS, DISPLAYING 1E+10.

>This should convince you that it is not correct to put 10000000000 in a >SINGLE variable. It will generate no error, but it is incorrect logically.

IT MIGHT BE LOGICALLY INCORRECT, BUT BY THE RULES OF THE "EDIT" PROGRAM, IT IS OK.

>Thus you cannot record 10000000000 in a single variable and conserve the >accuracy. It is the exact same as 10000000999. Exactly.

EXTERNALLY, YES, THEY DISPLAY THE SAME. BUT, INTERNALLY, AS PER THE "EDIT" PROGRAM, THEY ARE DIFFERENT.

HERE'S A LITTLE PROGRAM THAT SHOWS THAT:
1) THE NUMBER 10000000000 IS NOT ONLY VALID BUT ARITHMETIC CAN BE PERFORMED ON IT SUCCESSFULLY.
2) THE NUMBER 10000000999 IS INVALID, AND ARITHMETIC ON IT IS INCORRECT.

RUN WITH THE NUMBERS 10000000000 AND 10000000999 FOR RESULTS.
dim s as single
dim d as double
input s
print "as single ";s
d=s
print "as double ";d

d=s-1
print "-1 double ";d

Regards..... Moneo



 
 Respond to this message   

(Login burger2227)
R

*JEESH! USE THE IDE!

October 23 2007, 8:56 PM 


 
 Respond to this message   
Moneo
(no login)

I'm not ignoring you Ted,

October 24 2007, 6:16 PM 

it's just that I don't see how using the IDE is going to help determine if a number, input by the user, can be correctly expressed in a SINGLE variable.

If you think there is a way, please let me know.

Regards..... Moneo

 
 Respond to this message   

(Login burger2227)
R

*If you enter a double number QB IDE adds # automatically

October 24 2007, 6:28 PM 


 
 Respond to this message   
Moneo
(no login)

Fine, but what good is that if the number is assigned to a SINGLE?

October 24 2007, 7:34 PM 

What effect will it have? It obviously won't change my SINGLE variable to be a DOUBLE. So what happens? Will it hang due to overflow, or will it coerce the DOUBLE number (the one that it put the # on) to fit in the SINGLE variable?

All this happens at execution time. I can't see what the IDE's roll is during execution.

Regards..... Moneo

 
 Respond to this message   

(Login burger2227)
R

Well, if you recall, you did have problems

October 25 2007, 7:54 PM 

The IDE would have added the # to the end of the number and you would have known that if it is used with a SINGLE variable, it would be changed.

Just like your original post did! It may not be a big deal in most programs, but you were playing with numbers that were not SINGLE to stress your point about rounding with CINT.

Since you don't like to use the IDE, your solutions were wrong. I would HATE to have to compile something and TRY to guess the problems all of the time! The IDE is useful in that it could point out problems. Nobody is perfect!

Especially me LOL,

Ted

 
 Respond to this message   
Moneo
(no login)

Thanks, Ted,

October 26 2007, 6:50 PM 

I see your point about the IDE, but I'm still missing something.
Assuming the user was running my program using the IDE,
and he entered that big number into a variable defined as a SINGLE.
The IDE would add the # to the number,
but what would the user see happening?
Would there be an error condition visible to the user?
Or would the invalid input value still get stuffed into the SINGLE?

I'd check this myself, except I don't have a QBasic environment set up.

Thanks..... Moneo


 
 Respond to this message   

(Login burger2227)
R

INKEY$ a string and check the number of digits is 7.

October 26 2007, 11:28 PM 

I tried entering 12345678 and hit enter and it did not put the # after the number. It assumed it was a LONG Integer.

But then I added a decimal point to the end and it was replaced by the # .

The value usually is not that big a deal, but DOUBLE can only handle 15 digits. Then there is an overflow error!

Here is what I would do if you NEED a SINGLE (I love INKEY$):

LOCATE 10, 5: PRINT "Enter a decimal point number:"
dot = 0: num$ = ""
DO: n$ = INKEY$
IF n$ <> "" THEN 'required to use ASC
IF ASC(n$) > 47 AND ASC(n$) < 58 THEN num$ = num$ + n$: count = count + 1
IF n$ = "." THEN dot = dot + 1: num$ = num$ + n$ 'use if you want a decimal point number
IF dot = 2 THEN BEEP: num$ = "": dot = 0 'restart entry because of 2 dots
L = LEN(num$)
IF n$ = CHR$(8) AND L > 0 THEN num$ = LEFT$(num$, L -1) ' even allows for a backspace
LOCATE 10, 40: PRINT num$; SPACE$(8) 'show entry to user
END IF
LOOP UNTIL n$ = CHR$(13) OR count = 7 'for possible single number decimal point values only

IF dot = 1 THEN Svalue! = VAL(num$) ELSE Ivalue& = VAL(num$)

LOCATE 15, 10: PRINT "Single:"; Svalue!; "Integer:"; Ivalue&

You could do the same thing for DOUBLE values! Counting to 15 digits. The dot value is up to you, but a dot should be in all floating point numbers.
Otherwise they are Integers anyway. Integers would require maximum values to check for and may cause an error anyway.

Ted

END OF THREAD!

 
 Respond to this message   
Chandler
(no login)

How weird!

July 31 2005, 1:15 PM 

I was wondering about that!

 
 Respond to this message   
Moneo
(no login)

Rounding algorithms

April 27 2006, 7:43 PM 

Per Solitaire's suggestion in a previous post, please refer to the following Microsoft Knowledge Base Article on rounding:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;196652

This excellent article describes more than half a dozen rounding algorithms as applied to Microsoft products only, which leads me to believe that there are additional algorithms being used in the industry.

The most important point about implementing a rounding algorithm, is to first determine the precise algorithm required for this program or aplication. The user group specifying the requirements may not know the exact algorithm, and much less the correct name for it. Here is where the project manager has to explain the different algorithms in detail to the user group in order to arrive at a definition and then document it.

In Solitaire's post "Rounding Numbers and Round-off Errors", she offers us an excellent analysis of rounding issues. She goes on to provide the code for a function called RoundUp#. However, she somehow neglected to declare what rounding algorithm was being implemented. Using the name RoundUp#, and referring to the Microsoft article, we might assume that she was implementing the "Rounding Up" algorithm. But this algorithm has variations for SQL Server and Excel as well as for Visual Basic. So, which one did she choose to implement?

In an immediate reply to Solitaire's post, Earthborn posted a simplified version. However, Earthborn's version used the Symmetric Arithmetic Rounding algorithm. It just so happens that this is the most commonly used algorithm. My question is: How could a Symmetric Arithmetic Rounding algorithm be used as a simplification of a Rounding Up algorithm?

Earthborn's (Symmetric Arithmentic Rounding) code is indeed worth mentioning:
R = SGN(N)*(ABS(INT(N+.5)))

Slightly more straightforward:
R = SGN(N)*INT(ABS(N)+.5)

So, to wrap this up: decide on the rounding algorithm required, document it, implement it, and test it based on all the specifications of the algorithm, using positive and negative numbers, with odd and even values in the whole portion of the numbers, and decimal values of .0 , .1 , .5 and .9.

Happy rounding!
*****




 
 Respond to this message   
rpgfan3233
(Login rpgfan3233)

* Very useful, though FIX seems to be more reliable than INT in my experience.

April 27 2006, 7:55 PM 



----------
00100101 00110101 00110101 00100101 00110100 00110111 00100101 00110011 00111001 00100101 00110011 00110011 00100101 00110101 01100001 00100101 00110101 00111000 00100101 00110100 01100001 00100101 00110111 01100001 00100101 00110100 00111001 00100101 00110100 00110111 00100101 00110011 00111001 00100101 00110110 01100100 00100101 00110100 00111001 00100101 00110100 00110100 00100101 00110100 00111001 00100101 00110110 00110111 00100101 00110110 00110011 00100101 00110110 01100101 00100101 00110101 00110110 00100101 00110111 00110011 00100101 00110101 01100001 00100101 00110101 00110001 00100101 00110011 01100100 00100101 00110011 01100100 00100101 00110010 00110000 00100101 00110010 00110001

 
 Respond to this message   
Moneo
(no login)

Symmetric Arithmetric Rounding using FIX

April 28 2006, 4:34 PM 

I assume that you were referring to the algorithm for Symmetric Arithmetic Rounding as implemented in QB that appeared in my previous post. In that case, yes, FIX can also be used as well.

R = SGN(N)*INT(ABS(N)+.5) 'using INT

R = FIX(N+.5*SGN(N)) 'using FIX

For purposes of performing Symmetric Arithmetic Rounding, I've performed exhaustive tests on both of these implementations, and they work exactly the same. Neither is more reliable than the other. FIX simply truncates the decimal portion, and I must admit enables a cleaner algorithm. When INT is forced to work with positive numbers only, as above, then it too simply truncates the decimals.

Thanks for reminding me about using FIX.
*****


 
 Respond to this message   
Moneo
(no login)

Typo error in Microsoft Q196652 article

July 23 2006, 4:14 PM 

Rounding Up

......

Fix() rounds towards 0 (up in the absolute sense, but down in terms of absolute magnitude). Fix(-3.5) is -3.5.

NOTE: Fix(-3.5) is -3.5 should be: is -3.

*****

 
 Respond to this message   
Moneo
(no login)

Conclusions of July 2006 posts regarding rounding methods in QB.

July 24 2006, 4:52 PM 

First of all, I would like to thank all those who participated in the posts regarding rounding issues.

CONCLUSIONS:

Note: the names of the following referenced rounding methods were taken from the Microsoft Knowledge Base Article - 196652.

The most common QB functions used for rounding are:
* INT which does a form of rounding.
* FIX which truncates any decimal values.
* CINT and CLNG which do another form of rounding.

SYMMETRIC ARITHMETIC ROUNDING METHOD:
This is the most common and widely used method.
Symmetric means that positive and negative numbers of the same absolute value result in the same absolute result.
Algorithm in QB: (Proven algorithm)
R = SGN(N)*INT(ABS(N)+.5) 'using INT
or
R = FIX(N+.5*SGN(N)) 'using FIX

ASYMMETRIC ARITHMETIC ROUNDING METHOD:
Algorithm in QB:(May require further testing for full compliance)
R = INT(N+.5)

BANKER'S ROUNDING METHOD:
Algorithm in QB: (May require further testing for full compliance)
R = CINT(N)
or
R = CLNG(N)

The above algorithms were provided by QBasic Forum members.
Other rounding methods were not discussed.

For an unopinionated description of the rules for the above rounding methods, that is, what they are supposed to do, please refer to the above named document at:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;196652

*****

 
 Respond to this message   
Moneo
(no login)

Correction to Microsoft webpage address....

July 24 2006, 5:03 PM 

http://support.microsoft.com/default.aspx?scid=kb;EN-US;196652

Sorry.

 
 Respond to this message   

(Login emoneo)

Wooden nickels and QB rounding

November 29 2006, 12:21 PM 

"Don't take any wooden nickels"
and
don't assume what rounding method is used when the QB manual says that rounding is performed by certain operations.

Integer Division: The manual states: "Before integer division is performed, operands are rounded to integers or long integers..."
But, what rounding method is performed? Testing has revealed that the method used is the same as for the CINT function (see below).

The CINT function explicitly states that rounding is performed. We have found that the method is like that of Bankers Rounding.

The INT function does not state that rounding is performed, although it does not truly perform truncation like the FIX function does, since it returns the largest integer less than or equal. Variations of INT are used for rounding by incrementing/decrementing the number by ".5" before applying the INT. The most common of these is Symmetric Arithmetic Rounding.

Print Using: The manual states: "Numbers are rounded as necessary." Yes, but how are they rounded? Testing shows that the method used is Symmetric Arithmetic Rounding. Note: when rounding a negative number, like (-.4), Print Using will curiously display a "-0".

Note: All testing was performed with positive and negative numbers, as well as numbers with odd and even values in the whole part.

So, the point of all this is for programmers not to assume what rounding method is used when the manual for a given function says that rounding is performed. Do some testing first.

Regards..... Moneo





 
 Respond to this message   

(Login KristopherWindsor)

* '-0' is understandable; rounding shouldn't change the sign of a number. :-)

November 29 2006, 3:07 PM 



'Kristopher Windsor (geocities.com/tzmne)


Roses are red,
Violets are blue.
Sugar is nice,
And I'm lumpy, too.
- Don Wilson

 
 Respond to this message   

(Login emoneo)

That's a nice rule, but....

November 29 2006, 5:11 PM 

Using QB, if you take -.4 and round it using Integer Division (-.4\1) or round it usint INT or CINT, you will get zero with no sign. The only statement that will retain the minus sign is Print Using.

My quess is that all the other functions produce an integer result, and there is no such thing as minus zero on a two's-complement machine. On signed-magnitude machines, like the old IBM 7090, the registers had a dedicated sign-bit, so you could have a minus zero. On the other hand, the result of a Print Using is in a display format, which can express a minus zero.

Regards..... Moneo

 
 Respond to this message   

(Premier Login iorr5t)
Forum Owner

Ah, the IBM 7090!

November 30 2006, 6:20 AM 

Lots of blink-lights on the screen. Remember some old joke going around about "Watchen Der Blinken Lighten", meaning for newbies to stay away from the computer and don't touch any buttons?

Anyway, I found a use for -0. Remembering that memory was tight....

I was analyzing a program to reduce size. It had routines which would set condition bits in a 36-bit word. We thought we were clever to get 36 conditions recorded in a single word. Save memory, right?

Well, I noted that to set and clear and test the bits required 36 masks. Not to mention complicated tests (AND and OR, etc.).

So I changed the system to just say SSM in order to set a condition and SSP to clear, and so I used separate words for each condition (wasting, to the horror of my comrades, the other 35 bits). It actually saved memory. Rather than a couple of instructions to set a bit, one could just use SSM, setting the word to negative. And testing was a piece of cake: just BZE.

Luckily, zero was no exception. It set nicely to -0. If that were not the case I would need to waste some memory setting the conditions to some non-zero value. As it was, a simple SSP loop sufficed. Who cares what was in the other 35 bits? (Probably zero if the program was loaded fresh).

Those were the days...

Mac



 
 Respond to this message   
Moneo
(no login)

I hardly remember the 7090 and 7094 anymore....

December 1 2006, 4:57 PM 

The only assembler mneumonic I can remember is TIX (Transfer and Increment Index), where the increment was a specified variable. Powerful stuff!

I once did special things with the bits on the IBM 1401. You probably remember that each character had the 6 BCD bits plus a Wordmark bit and a parity bit. Well one time back in 1962, I had to validate the codes that were on input data cards that were to be written to mag tape. There were about 6500 valid codes. I got the 6500 valid codes on cards, read them into a 1000 character table, setting 7 corresponding bits (B A 8 4 2 1 Wordmark) at each table location. Then, when I read in the data cards I could check the table in a flash to see if the code was valid. I actually got the idea from someone showing me how to do direct table lookup on a 7090.

Ah yes, the good old days.....

Regards..... Moneo

 
 Respond to this message   
Current Topic - Rounding Numbers and Round-off Errors
  << Previous Topic | Next Topic >>Return to Index  
 Copyright © 1999-2014 Network54. All rights reserved.   Terms of Use   Privacy Statement  

Newbies usually go to www.qbasic.com and click on The QBasic Forum
Forum regulars have their own ways, which include The QBasic Community Forums